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Data in Many Forms... 


Statistics Canada disseminates data in a variety of 
forms. In addition to publications, both standard and 
special tabulations are offered on computer print- 
outs, microfiche and microfilm, and magnetic tapes. 
Maps and other geographic reference materials are 
available for some types of data. Direct access to 
aggregated information is possible through CANSIM, 
Statistics Canada's machine-readable data base and 
retrieval system. 


How to Obtain More Information 


Inquiries about this publication and related statis- 
tics or services should be directed to: 


Census and Household Survey Methods Division, 


Statistics Canada, Ottawa, K1A O16 (Telephone: 
990-9802) or to the Statistics Canada reference centre 
in: 


St. John's (772-4073) Sturgeon Falls (753-4888) 
Halifax (426-5331) Winnipeg (949-4020) 
Montréal (283-5725) Regina (359-5405) 
Ottawa (990-8116) Edmonton (420-3027) 
Toronto (966-6586) Vancouver (666-3691) 


Toll-free access is provided in all provinces and 
territories, for users who reside outside the local 
dialing area of any of the regional reference centres. 


Newfoundland and Labrador Zenith O-7037 

Nova Scotia, New Brunswick 

and Prince Edward Island 4-800-565-7192 

Quebec 1-800-361-2831 

Ontario {4-800-268-1151 

Manitoba 1-800-282-8006 

Saskatchewan 1(112) 800-667-3524: 
Alberta 1-800-222-6400 


British Columbia (South 
and Central) 112-800-663-1551 


Yukon and Northern B.C. 
(area served by 
NorthwesTel Inc.) Zenith 0-8913 


Northwest Territories 
(area served by 
NorthwesTtel Inc.) Zenith 2-2015 


How to Order Publications 


This and other Statistics Canada publications may be 
purchased from local authorized agents and other 
community bookstores, through the local Statistics 
Canada offices, or by mail order to Publication Sales 
and Services, Statistics Canada, Ottawa, K1A OT6. 


1(613)990-8114 
1(613)990-8115 
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PREF ACE 


Successful implementation of surveys demands, among other things, the design and 
development of. questionnaires which can be used to collect the required information as 
accurately and as cost-effectively as possible. However, the failure to devote sufficient 
attention, care and resources to questionnaire development has been surprisingly common in 
survey practice with many surveys failing to achieve their full potential as a conse- 
quence. This document, which provides a discussion with numerous illustrations of general 
principles, is intended to contribute to the promotion of good questionnaire design and 
serve as a reference and training tool. For the design of questionnaires on specific 
topics, the general principles discussed herein will need to be supplemented by subject- 
matter knowledge and by evaluation and analysis of past experience. 


Original material for the Development and Design of Survey Questionnaires was prepared 
in 1982 under contract to the United Nations Statistical Office in support of the National 
Household Survey Capability Programme. A review of the manuscript in Statistics Canada 
revealed that the conceptual framework was equally useful for survey designers in Canada. 
Accordingly the manuscript was revised to incorporate examples and illustrations drawn 
from Canadian experience. 


I wish to thank the United Nations Statistical Office for allowing this version of the 
manuscript to be prepared. In the preparation of the original version, the authors were 
assisted by Mukund Nargundkar and Don Royce whose contributions are hereby acknowledged. 


Finally, I wish to take this opportunity to pay special tribute to Simon Goldberg, 
former Coordinator of the National Household Survey Capability Programme, Director of the 
Statistical Office of the United Nations and Senior Assistant Chief Statistician of Canada 
whose untimely death in New York City on.May 24, 1985 has deprived the statistical com- 
munity of one of its most creative forces. It was Dr. Goldberg's belief in the ability of 
Statistics Canada to make a meaningful contribution in the area of questionnaire design 
which was instrumental in bringing this project to fruition. We hope that he would approve 
of this modest contribution to the art of survey taking. 


Martin B. Wilk, 
Chief Statistician of Canada. 
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INTRODUCTION 


Objectives and Audience 


The main purpose of this handbook is to 
discuss and illustrate basic principles in the 
development and design of questionnaires. 


It is primarily intended for survey design- 
ers and managers of surveys in government 
statistical organizations. 


Scope 


One of the most difficult problems in survey 
taking is the translation of complex ideas and 
concepts into usable questions. The variables 
to be measured by the survey have to be trans- 
formed into operational concepts and expressed 
as a logical series of questions which an 
interviewer can ask and a_ respondent’ can 
comprehend and answer. The structure and format 
of the questionnaire must facilitate the work 
of interviewers in asking questions and record- 
ing responses, and the work of coders’ and 
tabulators in capturing the data. 


The escalating demand for appropriate and 
timely statistical information of various kinds 
and from various sources calls for an organized 
approach to the process of data collection. One 
important element of an organized approach is 
the questionnaire which may be defined as a 
form containing a set of questions designed to 
elicit information upon a subject from a 
respondent. 


In the definition of questionnaire. are 
included administrative forms as well as census 
and sample survey questionnaires. 


Questionnaire development is a science and 
an art. It is a science to the extent that 
rules can be applied to the .formation of ques- 
tions which will elicit the information fully 
and accurately. It is an art because question- 
naires are based on language and it is diffi- 
cult to make language precise. Words often 
assume different shades of meaning in different 
parts of one country and more so in different 
parts of the world. Meanings also vary from 
one socio-economic group or age group to 
another. Different words or groups of words may 
be needed to stimulate the recall of the same 
event by different respondents. Thus, _ the 
success of a questionnaire depends to a great 
extent on the designer's ability to use 
language to convey to all respondents’ as 
consistent a request for information as possi- 
ble. 


Each survey presents new problems and pit- 
falls which must be anticipated and taken into 
account in developing questionnaires. 


Questionnaire development is not a labora- 
tory process. It is learned to a large extent 
by trial and error and perfected by practice. 


Content 


This handbook presents a number of chapters 
describing various elements in questionnaire 
design, development and administration. 


Questionnaire development. This chapter pro- 
vides an overview of the process of question- 
naire development and identifies the steps 
involved. 


Questionnaire design, production and adminis- 
tration. These chapters discuss general princi- 
ples of questionnaire design such as types of 
questions, wording, sequencing, formatting, 
precoding and placement of instructions on the 
questionnaire, and other issues of form and 
content which influence the ease with which 
information can be collected and processed. 


Testing and evaluation of the questionnaire. 
Testing of questionnaires eliminates problems 
related to design and interview and processing 
Situations, reducing response error and improv- 
ing overall management and control. 


This chapter describes a number of techniques 
for testing which include: reinterview follow- 
up; field visits by senior survey personnel for 
spot check and observation; interviewer de- 
briefing; experimental design studies, inter- 
penetrating samples; and comparisons with other 
data, such as administrative data, or data from 
other surveys and consistency checks. 


Checklist. Highlights recommended practices in 
questionnaire design and development in the 
form of a checklist. 


A Bibliography of useful sources for further 
reading, as well as reference is appended. 


The Questionnaire and Survey Design 


In most surveys, questionnaires ranging from 
lists of undefined topics to highly structured 
sets of questions are the key to standardizing 
and controlling the data collection process. 
Without specific question wordings and instruc- 


tions, respondents and interviewers inevitably 
change the meaning or emphasis of questions and 
bias the responses. 


The questionnaire plays a central role in a 
complex process whereby information is trans- 
ferred from those who have it to those who need 
itJs*or™ this’ transfer “tobe efrective,, =the 
questionnaire must meet the requirements of 
users and respondents. The expression of 
information needs, in terms suitable to the 
respondent is not something that can be accom- 
plished in one step. Instead, the questionnaire 
evolves and is refined as the survey develop- 
ment process unfolds. 


The process of questionnaire development 
consists of many elements such as_ content 
determination, question type and _ wording, 


sequencing and layout, 
etc. 


testing and evaluation, 


Figure 1 illustrates the questionnaire and 
the elements which together make up the survey 
design. Virtually any element could be placed 
at the centre of the network, but for the 
purpose of our discussion, we have chosen to 
focus attention on the questionnaire. 


Figure 1 
The Questionnaire and Survey Design 


Resources 


The 


Questionnaire 


In the actual design all elements inter-. 
relate to form a complex network which varies 
according to survey conditions and data re- 
quirenents. Changes to one element will require 
changes in others. For example, the ‘desired 
content often will be modified by considera- 
tions related to the resources, time, degree of 


reliability required, sample design, type of 
coding and processing. 

The questionnaire's total content, style of 
questioning;™ ‘format;) lengthy? etcs, ‘ares tale 


closely linked to the method of data collection 
and the survey's subject matter. Each method of 
data collection, creates its own ~- survey 
conditions to which a well-designed question- 
naire must be sensitive, through an appropriate 
style of questioning, content, format, length, 
and so on. The personal interview is often the 
only choice when a complex, long and demanding 
questionnaire is involved. In telephone inter- 
views much of the social interaction between 
interviewer and respondent is lost: the extra 
distance between interviewer and respondent may 
actually make it easier to answer sensitive 
questions because the respondent may feel less 


threatened, more at ease. In mail surveys, the 
questionnaire itself assumes the role of 
» 
a 
(w) 
Content 
Ms 
e 
S 
@ 


Data collection 


interviewer. It must introduce the _ survey, 
motivate the respondent to co-operate and guide 
the respondent in completing the interview. 


The questionnaire may also have to adapt to 
different collection methods: for example, in 
the Labour Force Survey the first interview is 
conducted in person while in urban areas most 
subsequent interviews are conducted by tele- 
phone. 


The heterogeneous nature of many survey 
populations frequently results in a need for 
cross-classified data. The degree of strati- 
fication affects the sample size, which in turn 
will affect the cost, timeliness, and amount of 
respondent burden which may legitimately be 
imposed. 


Particular designs of questionnaires affect 
data processing and its cost. The format of 
questions, open or closed, has direct implica- 
tions for operations such as coding, data 
capture, editing and tabulation. The presence 
of many open-ended narrative-type questions 
increases the time and effort required to code, 
and programs to edit and tabulate the data 
become more difficult and costly to write and 
Lest. 


Whether the survey is one-time or continuing 
affects questionnaire design. In continuing 
surveys, such as the Canadian Labour Force 
Survey, the same respondents are interviewed 
several times. Consequently, the questionnaire 
must take into account the total response 
burden during the respondent's stay in the 
survey. With a continuing survey, there is more 
scope for learning from experience and refining 
the questionnaire over time. It becomes impor- 
tant to document problems associated with 
particular questions. Experiments in question 
wording, programs to monitor response errors, 
and other methods of evaluating and improving 
the questionnaire design constitute an invalu- 
able source of learning for future purposes but 
are costly to conduct. The ability to improve 
the questionnaire must also be balanced against 
the disadvantages of change. The ability to 
make comparisons over time, the need to retrain 
interviewers, and the need to change expensive 
computer software must be taken into account 
when proposing changes in existing question- 
naires. 


While the questionnaire as an operational 
expression of user needs relates to the total 
survey design it neither fully determines the 
form of the other components nor is its form 
determined by the others. Questionnaire design 
flows from and is a part of the total survey 
design process. 
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Chapter 1 


THE PROCESS OF QUESTIONNAIRE DEVELOPMENT 


Introduction 


Surveys of various kinds can be a cost 
effective way of collecting information. With 
proper planning computers enable the production 
of survey results quickly, by relatively inex- 
perienced survey takers. Lack of planning how- 
ever leads to errors, the correction of which 
will consume inordinate amounts of limited 
budgets and delay the production of essential 
information. Every effort should be made to 
ensure that collection procedures and survey 
documents are as free of error as_ possible 
before being implemented in a field trial. 
Defining the responsibility for questionnaire 
design clearly, and at an early stage of survey 
planning is a first step in creating a mecha- 
nism to deal effectively with some of the 
issues which may arise during questionnaire 
development. 


Questionnaires can be designed by single 
individuals or by a team. In our opinion and 
experience, the team approach is particularly 
effective. Where questionnaire design must be 
assigned to a single individual the function 
and process remain essentially the same. What 
may be adversely affected is the efficiency or 
quality of the end product. In the discussion 
that follows, the use of a team approach is 
stressed. 


The process of questionnaire development 
aims to bring together designers and users in a 
coordinated and systematic fashion to specify 
the content of the questions, develop appro- 
priate instructions for interviewing, coding 
and editing, and establish the reports to be 
produced once the data have been cleaned of all 
collection and processing errors. 


Principal Steps in Questionnaire Development 


The. first, task ..for, the. designer .is.to, list 
all the activities involved in developing the 
questionnaire and estimate for each the kind of 
resources required. These activities should be 
organized into a logical time sequence, the end 
of which is fixed by the date for approval of 
the final questionnaire. Some activities can be 
carried out concurrently while others need to 
be done in sequence as the completion of one 
determines the beginning of another. To illus- 
trate, printing of questionnaires cannot start 
until specifications are finalized: that is, 
until the questions have been developed with 


answer spaces, codes, instructions, etc., and 
issues such as format, flow, colour, layout, 
etc., have been resolved. But, negotiations to 
find a printer, to set a tentative date for 
printing and to obtain estimates of costs can 
begin as soon as the decision to take a survey 
is made. 


The elapsed time to develop the question- 
naire should be planned to be somewhat greater 
than the real time required to accomplish the 
steps which must be carried out in a sequential 
fashion, but it may be shorter than the time 
required for doing all the steps. The plan 
should provide for unanticipated delays. If 
resources cannot be made available when needed, 
time frames will have to be adjusted. If time 
frames are not flexible, the outputs may be of 
lesser quality or scope. Thus, the designer 
ought to separate in his plan the steps which 
are essential from those which may be omitted 
without affecting the quality of the end pro- 
duct, should he run into scheduling problems. 


A general summary of the principal steps in 
questionnaire development follows: 


1. Decision makers consider a new program or 
policy or change to an existing program or 
policy. 


2. Program or policy analysts assess existing 
information, identifying gaps which require 
collection of data. 


3. A survey manager is assigned and he assigns 
responsibility for questionnaire design. 


4. Program/policy analysts describe their gen- 
eral plans and ideas with respect to: 


- expected uses of the data, 
- reports required. 


5. Questionnaire designer assists analyst to 
define the need for information in relation 
to specific objectives, subject matter area, 
specific populations, timing, etc. 


6. Questionnaire designer does a_ thorough 
review of the literature on existing designs 
and to determine what can be successfully 
modified and what needs a new design. 


7. Questionnaire designer schedules’ activi- 
ties and negotiates resources in terms of 
level, skills and availability. 


8. Questionnaire designer consults users to 


clarery: 


- major/minor objectives, 

- content areas of enquiry, 

- minimum quality, 

- mode of production of 
reports, tables), 

- sample size, 

- respondent nature and 

- language of interview, 

- field method, 

- data capture method. 


results (tape, 


characteristics, 


9. Questionnaire designer negotiates with 
field collection and data processing to 
provide input to interviewing and data 


capture plans and guides. 


10. Questionnaire design team starts drafting 
questions, testing them, recording the 
results and reporting back to the users. 


11. As questions are finalized including their 
codes and appropriate editing specifica- 
tions, the questionnaire designer arranges 
them into logical groupings. 


12. Questionnaire designer negotiates printing 
and production of the questionnaires and 
logistics of getting them into the field. 


13. Finally, the questionnaire is ready in a 
draft form with supporting instructions for 
the Pilot Test. 


14. The Pilot Test is conducted. The question- 
naire design team observes the training and 
the field work noting particularly what 
presents difficulties and the various ways 
interviewers resolve problems. 


15. The test questionnaires are subjected to 
all coding, editing, data capture and pro- 
cessing routines and reports are prepared 
on the quality of the resultant data. 


16. Final modifications to the questionnaires 
and supporting documentation are made by 
the designer and the team. Reports on test 
results are written and filed, along with 


examples of all documents used in the 
survey. 
17. The full-scale survey takes place. The 


results are analyzed. Observations of pro- 
blems and solutions are recorded and re- 
ports prepared to include the questionnaire 
designer's recommendations for future ques- 
tionnaire design. 


Working backward, the major critical dates 
for questionnaire design are: 


- date of start of field work: 
and documents must be finalized; 


questionnaire 


1Z 


- date of Pilot Test: a draft questionnaire and 
manuals are required; 

- the end of the period of consultation with 
users: after this there will be few oppor- 
tunities to make changes to the overall 
timing, budget and objectives of the survey, 
although minor modifications can be made, 
e.g., increases or decreases in sample size 
which do not influence major objectives of 
the survey. 


Organizing to Design, Test and Administer the 


Questionnaire 


A successful questionnaire design results 
from a number of skilled individuals working 
together to develop, test and administer the 
questionnaire. Ideally, therefore, a team will 
combine expertise in social and economic scien- 
ces with behavioural, mathematical and computer 
sciences. If resources are scarce the designer 
will have to carry out many functions himself. 
As much as possible he should consult others 
who have designed questionnaires in the past. 


Where a questionnaire design team is set up, 
each member should be given specific responsi- 
bilities. 


Questionnaire designer conducts the consulta- 
tion with users to determine what information 
is required; carries out a review of the 
literature and consults colleagues to obtain 
advice and assistance in designing the ques- 
tionnaire. The questionnaire designer usually 
comes from a subject matter area and generally 
has some background in the social sciences. His 
principal strength is the ability to understand 
the needs of users and to translate these into 
a coherent set of questions. 


Field expert provides technical support in the 
selection and training of interviewers, writing 
instructions, supervising assignments and 
resolving problems in the field. Generally the 
field expert is familiar with the characteris- 
tics of respondents. He can advise on the 
choice of respondents, and on wording questions 
so that respondents will understand them in a 
consistent fashion. The field expert will also 
be familiar with the peculiarities of language 
at the local level. His experience is of a 
practical nature and has been acquired in the 
field or in the delivery of programs. 


Data processing expert provides advice on the 
best layout of the questions and answer spaces 
for rapid and easy data capture, and on how to 
facilitate manual editing and coding in the 
fretd. 


His principal concern will be to structure the 
questionnaire so that the information can be 
captured efficiently to produce a data file 
which can be manipulated by users. 


All team members must possess’ sufficient 
independent authority to make proper design and 
implementation possible. In the design phase 
these experts will work collectively. Once 
field work begins, each expert will be respon- 
sible for his own area. 


In the course of questionnaire design, 
problems will arise and an efficient method 
needs to be developed for problem solving. Most 
disagreements can be resolved by relying on 
precedents or by referring the problem to other 
experts. "Content" or "Steering" Committees 
comprised of senior researchers or managers may 
usefully advise on content issues when users 
and designers fail to reach agreement. Often a 
Steering Committee will include representation 
from major user departments, creating a forum 
for discussion of issues and dissemination of 
EeESULES. 


Where there is little prior experience in 
questionnaire design, the solutions adopted may 
be regarded as precedents against which to 
resolve future problems. As a consequence the 
problem and the solutions to which it gives 
rise ought to be clearly and fully described 
and filed for reference purposes. 


Communicating with Users 


Communication with users will be more or 
less formal depending on the_ statistical 
environment in which questionnaire design takes 
places, highly .centralized; decentralized; or 
somewhere in between. 


Where statistical planning is part of cen- 
tral planning, the central statistical agency 
often has a determining role with respect to 
the production of data, and is represented as a 
matter of course on sector committees. The 
agency's statistical programs will reflect the 
priorities assigned to sectors by the central 
planners. 


Where the central statistical agency is 
located separately from central planning but 
with a mandate to coordinate the development of 
a national system of statistics, the agency may 
often create interdepartmental committees to 
obtain input from users. It generally sets its 
own priorities, recognizing the needs of 
various categories of users. 


Statistical agencies with a history of 
carrying out continuing national surveys have 
developed a number of ways of communicating 
with users. Communications with users may be 
unstructured, taking place as the need arises, 
or may be planned to include periodic conferen- 
ces with formal agendas fixed long in advance. 


Formal consultations are time consuming. 
Resources are required to prepare background 
papers for discussion, present them at meetings 
and produce proceedings. More than one round of 
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meetings may be required. The timetable for 
questionnaire design must allow sufficient time 
for this kind of consultation to take place. 


Communication is easier if standard recor- 
ding procedures are developed and applied. 
These could include the preparation and distri- 
bution of some or all of the following: 


- an "early warning memorandum" which 
advises senior officials of issues under 
discussion in user departments which may give 
rise to a need to coliect information; 


- a "proposal" form which describes’ the 
project as envisaged by users and designers; 


- an "approval in principle" form which, 
when signed by senior management, not only 
constitutes formal approval of a project by 
sponsoring divisions but also notifies 
functional areas that they may be asked to 


commit resources to the project at some 
future date; 

- a "questionnaire checklist" which ensures 
that questionnaires are reviewed by other 


interested divisions (outside the statistical 
organization or unit), and that their input 
has been obtained. 


These forms help stimulate internal discus- 
sion of the project, clarify the proposal for 
the purpose of communicating with users and 
sponsoring organizations. They constitute a 
permanent record of the planning which took 
place at the time of questionnaire design. 


Clarifying Content 


Texts on survey taking often state that the 
first task for the questionnaire designer or 
team is to determine the objectives of the 
survey. But very little is written about how to 
do this except to suggest that users be con- 
tacted early to canvass their needs prior to 
drafting questions. 


At the outset survey objectives are broadly 
stated, for example: 


- to provide a descriptive statistical back- 
ground for developing national policies and 
programs; 


on a continuing basis, and at 
some social or 


- to describe, 
national and regional levels, 
economic trends and phenomena; 


- to provide a general framework for more 
specific studies, policies and programs, etc. 


These broad statements of purpose need to be 
reduced to more manageable ones if they are to 
guide the questionnaire designer or the team in 
wording questions, preparing instructions for 
interviewers and designing the questionnaire 
for rapid data capture and processing. 


Redefining these broad statements of objec- 
tives into manageable forms suitable for a 
questionnaire is difficult to accomplish. And 
content priorities are easier to set if one 
knows exactly what time, money, and human 
resources are available for the survey opera- 
tions. 


On the whole, the questionnaire evolves and 
is refined as part of the overall survey 
development process. For example, users may 
want to plan a program of providing housing in 
a densely populated urban area and will need 
information on "housing". The typical user 
will approach the problem by asking himself the 
following questions: 


Q. What is the problem I am trying to resolve? 


A. Provide 
needs it. 


adequate housing to everyone who 


Q. What must I know to solve this problem? 


A. Present housing stock, demand for housing, 
production of housing and the impact of 
producing housing on other sectors such as 
transportation, utilities (such as water, 
electricity, sewer systems, etc.), educa- 
tion, recreation, etc. 


Q. How will I use the information once I get 
it, and how accurate and timely does the 
information have to be? 
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A. I'll write reports for expert committees; 
develop a program of financial assistance to 
purchasers; draft legislation to create an 
organization to deliver the program. 


In answering these questions the user's 
thinking becomes more quantitative. At this 
stage specific question wording is not an 
issue. The user is deciding what is to be 
measured, not how this should be done. Through- 
out this stage of clarifying and refining the 
information need, the questionnaire designer 
can assist users by advising them on the poten- 
tial difficulties of measuring certain items of 
information. It may be that local customs and 
values make the information too sensitive to 
collect; it may be that too many questions will 
be needed which will make field work too cost- 
ly; it may be that other items of information 
are more urgently needed and that this particu- 
lar request must be delayed; it may be that the 
information is already available and only its 
access needs to be negotiated. 


A useful technique to assist the question- 
naire designer to bring precision to the user's 
information need is to produce tabulation plans 
and dummy tables. Dummy tables are draft 
tabulations which include everything - titles, 
headings, column stubs - except the actual 
data. Production of such tables often indicates 
if the items of information to be collected can 
yield useable tabulations. Plans for all 
tabulations should be produced at the earliest 
stage possible. They will not only point out 
what is missing, but also reveal what is super-~ 
fluous. An example is given below. 


TABLE 1. Number of Household and Age-sex Composition of Population by Household Size 


Total 
number 
of house- 
Household holds Under 10-14 
size 10 
M F M F 
1 person 
2 persons 
3 " 
4 W 
5 " 
6 Ww 
Fh w 
8 " 
9 W 
LOR 
More than 10 
persons 


Total 


Number of persons by broad age groups and sex 


15-19 20-24 25-44 45-64 65 and All 
over 
pwr Merth Med Pp Miri MPasE Me te 


Examining dummy Table 1 reveals that the 
questionnaire must collect information on the 
number of persons in each household, including 
information on the age of each member so that 
they can be grouped by age, but not necessarily 


Single years of age, and on sex. The dummy 
table, however, does not indicate other 
variables such as if these persons must be 


related by marriage or blood, nor if boarders 
are. to be included or ' excluded. it 2s 
necessary to examine the concept of household 
and relate it to the purpose of the information 
collection before deciding to include _ or 
exclude persons. if the=~ imformation “as 
collected to estimate demand on public services 
SUCTMMEds Water, 9 CLECULICILY, “Stes, . 10° “TS 
important to know about all of the usual 
members of the household including domestics 
and permanent boarders. If, on the other hand, 
the objective is to know the size of the 
household whose members share their economic 
wealth, one may want to include all related 
family members including aunts, uncles’ and 
grandparents living together but not include 
boarders who do not share all their income with 
other members of the household. 


Institutionalizing the Capability 


Questionnaire design experience can be in- 
stitutionalized by establishing a questionnaire 
design capability and allocating resources to 
it on a continuing basis. While a formally 
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constituted group may start by designing 
questionnaires, with time and experience it can 
expand its role to include developing guide- 
lines, procedures and standards for question- 
naire design by others. 


Institutionalizing questionnaire design 
offers a number of potential benefits, some 
internal to the organization and_ others 


impacting more on the work outside. 


- improved cost-effectiveness and efficiency in 
internal questionnaire design; 


- improved quality of questionnaire design 
through development and = application of 
consistent standards; 


- questionnaires of predictably high quality 
leading to greater utilization of resultant 
data; 


- provision of assistance and advice to others 
in questionnaire design; 


- peer review as a mechanism to control expen- 
ditures on information collection. 
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Chapter 2 


QUESTIONNAIRE DESIGN 


Introduction 


An essential first step of questionnaire 
design is a well-defined statement of object- 
ives specifying the information required and 
its intended uses. While the process for reach- 
ing the objectives may require several rounds 
of discussions, at some point agreement will be 
general enough for the work of constructing a 
questionnaire to begin. 


The work of drafting the questionnaire, and 
in particular, the review of the kinds of 
results likely to be obtained, may lead to a 
reformulation of objectives. But regardless of 
how satisfied everyone is, the questionnaire 
designer or project team will be left with con- 
siderable freedom in deciding what questions to 
ask and how to ask them. 


This chapter discusses issues to be faced in 
the overall design of questionnaires: quality 
of data; grouping and ordering by priority of 
the topics to be covered in an individual sur- 
vey; the parts of a questionnaire, and features 
leading to well-designed parts; types of ques- 
tions and the advantages and limitations of 
each; wording problems; avoidance of "double- 
barrelled" questions (i.e., questions that are 
really two questions in one); limitations of 
hypothetical questions; and how to _ handle 
sensitive or threatening questions. 


Questionnaire and Data Quality 


There are no general rules to determine 
quality and reliability of survey data that 
apply to all surveys. For each survey, it is 
necessary to consider how the data will be used 
and how the survey results will be affected by 
errors of varying magnitude and kind. 


Experience shows that certain types of 
surveys produce more respondent burden and more 
non-sampling error than others. As these errors 
affect the quality of data derived from sur- 
veys, their control is an integral and vital 
part of questionnaire design. 


This section discusses the relationship of 
the questionnaire to a few of the more impor- 
tant sources of error and illustrates how the 
questionnaire can be designed to minimize 
error. 


Many forces motivate people to respond to 
surveys; an interest in the topic, a desire to 
be helpful, a belief in the importance of the 


survey, a feeling of duty, a belief in their 
own importance or even a belief that they have 
no choice but to co-operate. Other forces 
influence’ people to.” refuse; difficulty, in 
understanding questions, fear of strangers, the 
feeling of one's time being wasted, difficulty 
in recalling information, and embarrassment at 
personal questions. 


All of these forces will affect question- 
naire design through the way the questions are 
worded, the length of the questionnaire, and 
how survey topics are introduced. 


An appropriately designed questionnaire 
which helps respondents to respond will reduce 
error significantly. Unrealistic demands on the 
respondent's knowledge or memory, the use of 
overly difficult and technical language, or 
excessive demands on the respondent's patience 
are all causes of response error. 


Response errors may be systematic (response 
bias) or random (response variance). Questions 
on sensitive topics, such as amounts’ and 
sources of income, use of alcohol and tobacco, 
illegal activities or mental illness are sub- 
ject to large response errors: respondents may 
distort the answer to avoid embarrassment or to 
appear to conform to social norms (Warwick and 
Lininger (1975)). 


Questions which depend on the respondent to 
remember events, such as taking a trip or the 
occurrence of a crime, are another source of 
response error. Events may be forgotten, or 
incorrectly included. Bushery (1981), in an 
experiment with the U.S. National Crime Survey, 
found victimization rates per month with a 
three-month reference period much higher than 
those reported under a_ six-month reference 
period, which in turn were much higher than the 
victimization rates reported with a 12-month 
reference period. The bias due to recall loss 
with the longer reference periods was a much 
more serious source of error than sampling 
variability. The choice of an appropriate 
reference period for questions involving recall 
has been examined in a number of different 
subject matter areas (Sudman (1980), National 
Centre for Health Statistics (1973)). Bounded 
recall (see also Chapter 2, "Built-in Assump- 
tions") where respondents are interviewed at 
the beginning and the end of the reference 
period, and the use of prominent dates (e.g., 
Christmas) or calendar aids to jog respondents' 
memories have shown to be of some value in 
reducing under-reporting (Neter and Waksberg 
(1965), Ashraf (1975)). 


Response errors may also result from the way 
respondents interpret and answer questions, and 
the way in which interviewers interpret and 
record answers. The interview is a dynamic 
interactive process of communication between 
the interviewer and the respondent. How it is 
handled determines whether it will produce the 
desired information in an accurate and effi- 
cient fashion. It is the questionnaire, through 
its content, question wording, instructions and 
layout, which must play a major role in con- 
trolling the situation. 


Once the interview is completed, the ques- 
tionnaire becomes primarily a data processing 


document. Errors can occur at all phases of 
processing including coding, data capture, 
editing, imputation, estimation and _ tabula- 
tion. The way in which the questionnaire is 


designed will have a significant impact on the 
number and type of errors at this stage of the 
survey. By including codes right on the ques- 
tionnaire, for example, data capture errors are 
usually reduced significantly. The data are 
captured directly from the questionnaire with- 
out first being transcribed onto another form. 


Problems of missing or inconsistent data can 
often be traced back to faulty questionnaire 
design. The ability to reconstruct or impute 
for missing values often depends on other vari- 
ables or fail-safe procedures included in the 
questionnaire. For example, in a survey which 
requests information on several detailed compo- 
nents of income, cases where the information is 
not given or is incorrect can often be salvaged 
by including a question asking for total 
income. 


Response and data processing errors are a 
Few of the non-sampling errors which are close- 
ly linked to the questionnaire. It must be 
designed to minimize these as far as possible. 
An important process in minimizing and con- 
trolling the non-sampling error is testing of 
various components of the questionnaire at the 
design stage. Subdsequent chapters discuss 
various procedures and alternatives that may be 
undertaken before the final version of the 
questionnaire has been decided on. 


Grouping Subjects 


In an ongoing program of statistics, data 
may be wanted on a great number of subjects - 


employment, income, expenditure, education, 
health, agricultural production, housing, 
energy, and so on. A number of factors should 


be considered if this is the case. Apart from 
user needs, legislative requirements, and scope 
and timing, a major factor will be the com- 
patibility of different subjects. For example, 
in expenditure surveys, the scope of the data 
wanted may require so much taxing of memory, 
and assembling of records, it may be unwise 
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to burden the same respondents with detailed 
questions on income beyond those needed to - 
arrive at a rough total figure for balancing 
against reported expenditures. Some of the 
topics to be covered in a survey, may be 
incompatible with others. For example, detailed 
expenditure questions followed by questions on 
income may result in biased reports of income 
if tax has been evaded. 


Although the aim may be to divide subjects 
into separate non-overlapping units, each of 
which will become the object of a separate 
survey, there can be some advantages’ to 
including a degree of overlap or repetition. In 
a program of continuing surveys, it becomes 
useful to establish a core of demographic 
questions that will be the same in each sur- 
vey. It will then be possible to make com- 
parisons between surveys, and to carry out 
comparative analyses between subjects. It may 
also be worth considering limited repetition of 
non-demographic questions aS a= means_ of 
checking the reliability of data obtained on a 
given subject. 


While a logical or natural grouping of 
subjects is desirable, the questionnaire 
designer may have no choice but to combine 
seemingly incompatible subject matters and 


write instructions for a heterogenerous group 
of interviewers. 


The designer will want to keep these consid- 
erations in mind in writing instructions to 
interviewers and in limiting questions = and 
planning their flow. 


Once selected, topics must be given an order 
and a priority. For example, in a survey on 
health, topics could include physical and 
mental health, the nature and seriousness of 
ill-health, duration and type of treatment 
sought and received, effects on employment or 
income, effects on the health, employment, 
etc. of others, illness-prevention practices of 
the household, use of health facilities and so 
on. Some of these data may be obtainable from 
administrative sources (e.g., hospital records) 
although linkage with other data obtained from 
household survey respondents may be difficult. 


Usually, it is the questionnaire designer, 
in consultation with users, who will set 
priorities for topics, and for sub-topics and 
determine the amount of detail to be sought on 
each. 


There are advantages to a division of the 
general topic of the questionnaire into a 
series of sub-topics. 


- it is easier to develop one section, 
small number of sections, at a time, 
than all at once; 


Orbea 
rather 


- it may be feasible to test one section, or a 
small number of sections, at a time, rather 
than wait for all sections or sub-topics of 
the questionnaire to be developed; 


- if resources are available sections can be 
developed separately, by different designers 
or design teams, speeding up the development 
process; 


- decisions on the flow of the questionnaire 
can be made more readily, as it becomes 
easier to consider the effects of different 
orders of sections when they are considered 
as separate. 


Making Concepts Operational 


While topics within a survey are being dis- 
cussed, a number of concepts will have been 
considered. These concepts will later need to 
be turned into questions in a questionnaire. 
Terms must be defined and applied the same way 
in the questionnaire, the interviewer manuals 
and the instructions given to respondents, and 
in reports based on the data. This will ensure 
that the precise meaning wanted is conveyed at 
all times. 


Where information from one round of survey 
taking is compared with information in a _ sub- 
sequent round, the concepts must withstand the 
test of time and the words to describe the 
concepts will need to be chosen with care. In 
fact many questions may be needed to convey the 
correct meaning. If terms such as "household" 
or "income" or "ill-health" or "production" are 
not defined, interviewers and respondents will 
interpret them in various ways. Because of the 
mixture of meanings, totals or averages from 
the data may be meaningless. 


As an example, to some, unemployment means 
working less than full capacity, to others it 
means not working when one has an opportunity 
to do so, in other words refusing a job. Or 
unemployment may mean not working at the job 
for which one has been trained. In the Canadian 
Labour Force Survey unemployment means none of 
these. Jo Labour Force Survey statisticians, 
"unemployed" includes those persons in_ the 
Civilian (non-military) non-institutional popu- 
lation, aged 15 and over who during the ref- 
erence period (a given week each month): 


- were without work, had actively looked for 
work in the past four weeks (ending with the 


reference week), and were available for 
work. The "available" are those (including 
full-time students seeking part-time work) 


saying that there was no reason why they 
could not take a job in the reference week, 
Or, ihe they ‘could “not™ take ~a job it was 
because of "own illness or disability", 
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"Dersonal or family responsibilities", or 
that they already had a job. 


- (until 1984) had not actively looked for work 
in the past four weeks, but had been laid off 
(from a job to which they expect to return) 
for 26 weeks or less and were available for 
work (in 1984 these individuals were re- 
classed as “outside the labour force"); or 


- had not actively looked for work in the past 
four weeks but had a new job to start in four 
weeks or less from tne reference week, and 
were available for work. 


The "institutional" population, excluded 
from the definition, comprises those living in 
prisons, mental hospitals, long-term disability 
institutions, and the like. Up to nine ques- 
tions must be asked to ensure that the correct 
meaning and only that meaning is conveyed to 
respondents. 


Questions should be asked in a manner that 
will be understood by the majority of respon- 
dents. As a general rule it is preferable to 
use words with which the majority of respon- 
dents will be familiar and continue to be fami- 
Ttar'. 


In a continuing survey program, only those 
changes which have been shown to be necessary 
should be made. Often special studies of the 
continued usefulness of concepts and wordings 
will assist questionnaire designers to decide 
about changing concepts. Because of their 
interest in producing data for trend analysis, 
questionnaire designers should generally be 
conservative towards the application of new 
concepts - arguing that concepts should reflect 
change, not precede it. 


Besides the choice of words, the degree of 
detail must be considered at the time concepts 
are being turned into usable words and phrases 
for questionnaires. For example, if income data 
are wanted, is it enough to ask for a total 
figure of income from all sources, before (or 
after) taxes, and for a specified time period? 
Or should a detailed breakdown of income be 
obtained from each of tt following: 


- wages and salaries for each job held during 
tne reherence: perrod; 

- sales ser- 

Valcese 


of household-produced goods or 


- benefits or allowances from the government 
(and perhaps from each of several government 
agencies) ; 

- cash benefits or allowances’ from 

companies or organizations; 


private 


- gifts from private individuals; 


- income from investments, loans, bank depo- 


sits; 


income received in kind (i.e., as merchandise 
or services), given as an estimate of the 
market value or what would have had to be 
paid if the items had been bought; 


benefits or allowances in kind from companies 
or organizations (such as, the use of equip- 
inent, houses, and so on, at no charge or at a 
charge less than the market value). 


Similarly for family expenditure surveys - 
should expenditures be asked on, all clothing, 
or by detailed list of clothing items. Clearly 
such factors as the objectives for which the 
data are being sought, the likely extent of 
error (e.g., under-reporting for a "total 
income" or ‘rocalt- ec lothing expenditure" 
question), and the burden on respondent time 


a od Statistics Canada Statistique Canada 


| 


SELECTED MANUFACTURING OUTPUTS 


Manufacturing and Primary Industries Division 
ANNUAL CENSUS OF MANUFACTURES, 1984 


ing this questionnaire please 
quote this 7 digit number. 


> a 


| In all correspondence concern- 
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and memories in asking the itemized questions, 
must be considered in the development of usable 
questions from the concepts to be included. 


Questionnaires vs. Schedules 


FOUNDATION GARMENT INDUSTRY 


Once a general organization plan has been 


established, attention must turn to the 
individual questions to be asked. At this 
stage, a choice is available as to the degree 


of completeness of wording to be provided to 
interviewers. 


It is . usually, - desirable. ,topseprovide 
interviewers and respondents with as exact a 
wording as possible for any given question. 
Only then can the questionnaire designer be 
sure that each respondent has received the same 
question stimulus. In the absence of exact 
wording answers may differ primarily because of 
differences in wordings used by interviewers. 


Si vous préférez recevoir ce questionnaire 
en francais, veuillez cocher la case et 
retourmer a la Division des industries 

manufacturiéres et primaires, 
LJ 


Statistique Canada 
Ottawa, K1A 0T6 
Physical Location of Etablishment (Please correct if necessary ) 


8. Shipments of goods of own manufacture (refer to Instruction 8 in the Reporting Guide) 
BASIS OF VALUATION OF SHIPMENTS Net value of shipments, 
Yy N excluding sales taxes, 
& - excise duties and 
8.0.2 Are you reporting shipments as requested? 1 L] 1) Commodity } excise taxes, shipping 
Ye N aaa Hae Unit charges by common or 
= iS eiietics of Quantity shipped contract carriers 
8.0.3 Are reported Values at Final Selling Price? 1 2 eal Capadawee measure and net of any 
If ‘‘No” please specify basis of valuation: sales discounts, 
allowances, etc. 
Canadian dollars 
8.0.4 | (omit cents) 
8.1 Products shipped 
Sl leiGirdiles, allity pess(excludinespantic-girdles)j ens . <u. ey erie cee 788 71 i dozen all 
eller lees CMOS, AUN TCM = yg ten ale os gue -e 6 ome oo aiid ol anad oluie 788 72 j 
788 73 § 
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Occasionally, the circumstances of data 
collection are such that use of exact wording, 
is impractical. Sometimes it may even be detri- 
mental to the quality of response. 

For example, it may not be practical to 
find, hire and train interviewers to use 
unvarying wording. As well, interviewers may 
not feel comfortable reading a text developed 
by others, one that does not correspond to 
their own pattern of speech, or allow for their 
own pauses and intonations. It may be desirable 
for the interviewer to be at ease and to speak 
in a natural-sounding manner. Depending on the 
characteristics of respondents, and the nature 
of the survey it may be better to provide no 
more than a summary of the important points 
that should be conveyed to respondents and 
allow interviewers to use their own words. 


Local customs or culture may make respon- 
dents suspicious of any stranger, or even a 
local person working for the government. In 
this situation, it may be of benefit to allow 
interviewers considerable flexibility in impro- 
vising the wording of questions. There may be 
social, religious, or traditional beliefs such 
that abrupt direct questions would lead to 
biased answers, outright evasion, or _ even 
hostility. In these conditions, great reliance 
must be placed on the skill of interviewers to 
develop rapport with respondents through, a 
gradual leading-up to questions in a conversa- 
tional style. No questionnaire could ever 
provide the exact wording an interviewer should 
use, or all the questions and comments needed 
to get a good answer. 


Under interviewing conditions described in 
the last few paragraphs, it may be better to 
use a schedule type of form rather than a 
detailed questionnaire. By schedule is meant a 
form that does not give the exact wording to be 
used, but gives instead a summary of the infor- 
mation to be obtained. Often the summary 
descriptions look like column headings in a 
tabulation, e.g., "“education-level" instead of 
"what is the highest level of schooling you 
have completed?" or "how many years of school- 
ing have you completed?"; "number of children 
ever borne" instead of "how many children have 
you ever borne?" 


While the questionnaire approach emphasizes 
the need for interviewers to use the wording as 
given, without changing it, the schedule 
approach emphasizes interviewer/respondent 
understanding of the concept that each question 
is to convey, to minimize bias that might come 
about because of differences in interpretation 
between interviewers. A schedule approach can 
work satisfactorily where there are _ well- 
trained interviewers/knowledgeable respondents. 
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The schedule approach cuts down on the bulk 
of the questionnaire and is most useful where 
limitations on availability of paper or print- 
ing facilities are significant, or where inter- 
viewers or field supervisors are required to 
carry questionnaires for long distances under 
poor transportation conditions. 


Most self-completion business survey ques- 
tionnaires use a schedule approach, in which 
column headings correspond closely to the way 
in which business records are kept. This prin- 
ciple is illustrated in the example above drawn 


from the 1984 Census of Manufacturers - 
Foundation Garment Industry. 
Wording of Questions 

The wording of schedule questions is 


generally limited to a summary description of 
the information to be obtained. Typically, the 
questions are "age", "education level", 
"occupation", "number of acres" and so on. 
Provided interviewers are informed thoroughly, 
by training and in their manuals, of all inter- 
pretations and shades of meaning that could 


apply, the exact wording used may not be very 
eritieal. 
Where questions are worded completely, a 


number of aspects of the wording must be con- 
sidered. The clarity and precision of the words 
used, biases in particular words or phrases, or 
as a result of the order of the words, or the 
degree of technical or other knowledge they 
presume respondents to have, must all be taken 
into account. It is all too easy to think that 
one can draft a perfectly worded questionnaire 
while sitting in an office. In fact, it is very 
difficult to imagine all the possible interpre- 
tations and the variety of answers respondents 
may give, or the different circumstances or 
conditions which may alter the sense of the 
questions. 


A questionnaire designer should design ques- 
tionnaires only after a period of experimental 
data collection. This means going out into the 
field and interviewing or observing inter- 
viewing. It is important to become familiar 
with speech patterns, vocabularies, and ways of 
thinking of the respondent population. 


The questionnaire designer also should know 
the characteristics of the interviewers for 
whose use he is designing questionnaires. He 
will need to be aware of interviewers’ methods 
of work, the shortcuts they may take, how they 
are paid (per hour or day, or per completed 
interview) and how that may affect their work. 


A technique for question development that 
might be tried if resources permit is to call 
on staff members or professional interviewers 
to word their own questions, after a discussion 
of the survey objectives. Then select five to 
eight excellent interviewers and instruct them 
to go out for one day and interview people. The 
next day discuss their interviews concentrating 
on areas where the replies were incomplete or 
not relevant to the objectives. Repeat this 
process several times, each time sending the 
interviewers out with more standardized ver- 
sions of the questions. Each time there will be 
fewer alternatives to consider and eventually 
agreement will be reached about how to ask the 
questions. By this time enough responses will 
probably have been obtained to begin to evalu- 
ate them in light of tabulation plans. Ques- 
tionnaires incorporating questions developed 
this way should still be tested, for example, 
to see whether average interviewers can carry 
out the interview. 


Clarity, Precision and Choice of Words 


The language of a question should be 
simple. Words used should be familiar to 
respondents yet appropriate for the survey 
subject matter. The kind of vigorous language 
generally used by journalists could be worth 
aiming for. It is important to choose precise 
words carefully, avoiding vagueness and ambi- 


guity. Compromises may be required between 
accuracy and length. 
Consider the question: "What is your 


income?" “Does "your" refer tothe respondent or 
to his family or household?; what is the time 
period - last week? last month? the last 12 
months before interviewing?; what "1s to” be 
included in "income" - salary and wages? tips? 
piece-work pay? income from other’ sources? 
income in kind? How should this non-monetary 
income be valued? For farmers, is this the 
total income of the farm, or just the "profit" 
left over after all expenses of running the 
farm have been taken care of? 


"household" can lend itself to a 
variety of interpretations, depending on the 
culture, social arrangements, housing avail- 
able, and so on. 


The term 


Belson (1981) gives a number of examples of 
widespread variations in understanding, or 
outright misunderstanding, of everyday words 
such as "usually", "have", "weekday", "chil- 
dren", "generally", "regularly", and so on. 


In a U.S. survey, respondents were shown a 
bottle of orange drink and asked: 
"How much orange juice do you think it 
contains?" 


The following answers given by a number of 
respondents illustrate the kinds of answers 
that were given to such an imprecise question: 


Tie 


one orange and a little water and sugar 
25% orange and 75% carbonated water 
juice of one-half dozen oranges 

3 ounces of orange juice 

full-strength 

a quarter cup of orange juice 

none 

not much 

a small amount of orange juice 
one-fourth orange juice 

doubt it 

don't know 

not very much 

3 to 4 ounces of orange juice 

a pint 

most of it 

a little water mixed with orange juice 
about a glass and a half. 


Better ways to word this last example might 
have been: 


"This bottle holds 16 ounces of a drink. How 
many ounces of that would you say is orange 
juicer! 


"What percentage of this drink would you say is 
orange juice?" 


"What part of this drink - a quarter, a half, 


three-quarters, or what - would you say is 
orange juice?" 
Examples are often ‘useful, to clarifyua 


concept. Where they are used it is desirable to 
give more than one and to make them as neutral 
as possible. Otherwise, respondents may have 
trouble understanding the underlying concept. 
They will tend to be misled by thinking just of 
the single example, rather than the class or 
group from which the example has been taken. 


Where it is necessary to use technical or 
legal terms, definitions or explanations should 
be provided. Otherwise, respondents may not be 
sure if they are aware of or have had experi- 
ence with the concept. They may guess, or claim 
awareness or experience more than is true in 
the researcher's terms. 


Order Bias 


When respondents are asked to choose among a 
number of alternatives that have no logical 
order, a pattern of answers may emerge that is 
the result of the order in which the alterna- 
tives are presented rather than their impor- 
tance or relevance to the respondent. While the 
bias is most noticeable in opinion questions, 
it applies equally to supposedly factual ques- 
tions. Lists of activities, or of items or 
articles purchased, owned or used, from which 
respondents are expected to select those that 
apply to them, can all suffer from the bias. 


In an example from Payne (1951) several 
alternatives, A, B, C and D, were presented in 
different orders, to matching samples’ of 
Fespondents. Each -one¥ of), A.B, 5.C. Dj, iwas 
selected more often when it was at extreme ends 
of the list than when it appeared near the 
centre. 


Alternative A was selected by: 


27% when it appeared at the top of the list; 
17% when it appeared near the centre; and 
23% when it appeared at the bottom of the list. 


Alternative B was selected by 


11% when at the top; 
*° When near the centre; and 
7 when at the bottom. 


A very similar pattern of responding was 
obtained for alternatives C and D. 


This sort of situation is quite common. When 
a series of alternatives is read out, some 
respondents will find it easiest to remember 
the first, others tend to remember the last 
one. The items in the middle terd to lose out, 
either way. With an ordered set, such as a set 
of numerical values, respondents will tend to 
choose values in the middle (see Chevry, p. 
136)... 


One solution to this problem is to present 
alternatives in all possible combinations, to 
as many randomly picked groups of respondents 
as there are orders. This will help balance out 
some of the bias due to ordering. But this can 
cause practical problems, such as requiring 
more than one version of the questionnaire, 
with consequent control problems in the field- 
work. Separate versions of the questionnaire 
would also need to be clearly identified to 
avoid confusion during capture of the data. It 
can also mean a large number of alternative 
versions - with three items, there are six 
possible combinations; with four items, there 
are 24, and so on. However, with some compromi- 
Sing and careful ordering, a practical smaller 
number can usually be found. Here is an example 
of how one question with a variety of choices 
and orderings, could be handled. 


Which of these crops would you say gives the 
greatest profit to most farmers? (SHOW CARD) 


Card A Card B Card C 
Wheat Corn Potatoes 
Potatoes Wheat Wheat 
Corn Potatoes Barley 
Barley Barley Corn 
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In this selection, each crop appears once at 
an extreme end of the list (either first or 
last), and once in 2nd or 3rd position. As 
well, no two crops appear together twice. One 
third of respondents would see each card. 


Another possibility is to mark question- 
naires in the office before they are shipped to 
interviewers to show the item to be read first 
in each successive interview. Care will need to 
be taken to provide each interviewer with an 
equal number of questionnaires marked to start 
with each item. Alternatively, interviewers 
could be instructed to change the order of men- 
tion of the list of items by starting with the 
second item in the second interview, and ending 
the list with the first item. Then the third 
interview starts with item No. 3 and ends with 
item No. 23; and so on. Interviewers need to 
keep careful track of how many interviews they 
have completed, to keep the rotation order 
correct. There is a greater possibility of 
confusion and error with this method, since 
interviewers are already busy enough in a 
complex task. 


It may also be possible to split the number 
of items into two (or more) groups. A respon- 
dent would be asked about each group sepa- 
rately; then a second question would ask about 
the preferred one of those marked first or most 
preferred from the preceding set of questions. 


Social Desirability and Politeness Biases 


Respondents will tend to choose answers that 
are most favourable to their self-esteem. They 
will also tend to choose answers that’ they 
think make them look intelligent or thoughtful 
or are in accord with social norms. In some 
cases there is a general desire to be polite 
and cooperative with interviewers and to slant 
answers towards the kind of answer respondents 
think interviewers would like to hear. This 
bias can be particularly strong if interviewers 
are perceived as powerful or _ threatening 
authority figures. In other cases non-response 
will increase. 


Because of social desirability respondents 
often exaggerate their possessions, claim to be 
richer than they are, or lie about their age. 
They may claim more awareness or knowledge of 
something than they really have; or they may 
claim certain behaviour patterns that are not 
real. The history of birth control studies is 
filled with examples of spurious results from 
respondent politeness bias, in which, for 
example, an exaggeratedly high use of contra- 
ceptives was claimed. 


Many studies have obtained high levels of 
claimed awareness and use of non-existent pro- 
ducts and services; sometimes it can be useful 
to include such dummy items in a list just in 
order to measure the extent of spurious aware- 
ness or use that may be claimed for real pro- 
ducts. 


Family expenditure data from household sur- 
veys in the United States and in Canada consis- 
tently show reported expenditure on alcohol at 
about one half of actual industry sales. Sur- 
veys about alcohol consumption, followed by 
surreptitious examination of garbage put out 
for disposal, have also suggested gross under- 
reporting. 


Regular surveys to measure the proportions 
of smokers in the general North American popu- 
lation have been registering a gradual decline 
during the last few years, although industry 
sales have been increasing. The decreased 
social acceptability of smoking, so that some 
smokers claim not to smoke, may be an element 
in the drop in the reported number of smokers. 


The bias effects can be reduced, although 
never entirely eliminated, by minimizing the 
desirability of one response over any other, 
for example, by suggesting as a lead-in to the 
question that undesirable answers as well as 
desirable ones have been given by other 
respondents (e.g., "Many people have said... 
while others have said...What has been your 
experience?"),. 


A related bias is the wishful thinking that 
can occur if respondents are asked about their 
future behaviour plans, or are asked about 
hypothetical situations ("what would you do 
if...?"). In general these sorts of questions 
should be avoided, as they seldom yield answers 
indicative of the actual behaviour once the 
time-period is over, or when the hypothetical 
Situation in fact comes to pass. 


For subjects that arouse strong feelings 
among respondents, the choice of words and the 
biases (or at least differences in results) 
that emerge can be quite striking. Clearly the 
choice of words in such questions must be made 
very carefully. 


A U.S. poll in August 1980 asked a random 
sample of Americans about their views of abor- 
tion in several different ways, and obtained 
several results. 


First, respondents were asked, "Do you think 
there should be an amendment to the Constitu- 
tion prohibiting abortions, or shouldn't there 
be such an amendment?" 
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The respondents were solidly opposed to such 
an amendment. Only 29% of those surveyed 
favoured the proposal, while 62% were opposed © 
and the rest were uncertain. 


Later, the same people were asked a differ- 
ent question to measure attitudes on the same 
issue. It is the preferred question of the 
anti-abortion movement, and the results 
indicate why this group likes it so much. 


The question read: "Do you believe there 
should be an amendment to the Constitution pro- 
tecting the life of the unborn child, or 
shouldn't there be such an amendment?" 


The result: 50% in favour, 39% opposed, the 
rest undecided. - 

About one fifth of those polled shifted 
position when the second question was posed. 
Fully one third of those who opposed the amend- 
ment when it was presented as "prohibiting 
abortions" supported it when it was presented 
as "protecting the life of the unborn child". 


Among those who have a clear view - those 
who took a consistent position on the anti- 
abortion amendment however the question was 


phrased - those opposing the measure had the 
edge. 

Over all, 36% of the respondents consis- 
tently opposed the amendment, while 26% 
consistently favoured it. The rest’ either 


Switched from one side to the other or were 
undecided on one or both phrasings of the ques- 
tion. Those with lower levels of education 
tended to be more easily swayed by question 
wording, but nearly one in six among the well- 
educated respondents were also influenced by 
wording. 


On another question in 
anti-abortion respondents 
minority. 


the same poll, the 
were again in the 


Those polled were asked: "If a woman wants 
to have an abortion and her doctor agrees to 
it, should she be allowed to have an abortion 
or not?" 


Sixty-two per cent said a woman should be 
allowed to have an abortion; 15% said it depen- 
ded on the circumstances; only 19% said a woman 
should not be allowed to have an abortion. 


Double-barrelled Questions 


Double-barrelled questions or questions 
which have two or more questions "nested" 
within them should be avoided. Respondents 


become confused in trying to answer the ques- 
tion, especially when they have’ different 
answers for each part. 


One indicator of. ¢the! eidkelithoods\ lof»sa 
"double-barrelled" question is the appearance 
of the conjunction "and" or "or" in the ques- 
tion. The best way to avoid confusion is to 
replace double questions containing "ands" or 
“or's" with two or more single questions. 


Explanation of Technical Terms 


It is useful to give an explanation or defi- 
nition prior to utilizing a term that has a 
technical sense, especially where the term is a 
common one with a variety of non-technical or 
imprecise meanings. Here the questionnaire 
designer will want as much as possible to 
ensure that the term is understood by everybody 
in the sane technical sense. If the term is 
used without being defined, or followed by the 
definition, the respondent may mentally frame 
his answer to the question before the explana- 
tion is given. In so doing he may answer incor- 
rectly using his own definition rather than the 
one provided by the researcher. Example: "where 
were you living when Old City and New City 
joined together to become one city - that is, 
amalgamated?" would be better than "...when Old 
City and New City amalgamated?", confusing res- 
pondents who don't know what the word means. 


Built-in Assumptions 


It is desirable to make sure a question is 
applicable to respondents who encounter it. 
Difficulties can be avoided by proper lead-in 
questions which determine whether the respon- 
dent is qualified to answer it. For example, 
the question "How old is your wife?" should be 
preceded by questions that determine: (a) that 
the respondent is a male of marriageable age, 
(b) that he admits to having a wife, and (c) 
which wife is being referred to. 


Questions should not assume that the respon- 
dent has knowledge or awareness in very 
specific areas. Such questions could embarrass 
or even annoy some respondents while others 
will claim knowledge they don't have, so as not 
to look foolish. Here is an example of a 
question that would not be suitable to ask the 
general public: "Please tell me all you know 
about the migration policies of the government 
of the Republic of San Marino". 


The questionnaire for the 1946 Census of 
France included two questions: "Can you read?" 
and "Can you write?". An unusually high propor- 
tion of negative answers were received in 
Alsace, a region of France that is largely 
German speaking; a follow-up survey determined 
that many respondents had understood’ the 
question as referring to reading and writing 
French (Chevry, p.. 121). 
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A 1958 housing survey among miners and 
steelworkers in several European’ countries 
asked "how many rooms (including the kitchen) 
do you occupy?" Some respondents, with fami- 
lies, understood the question as meaning "you 
personally" and answered "one" although the 
intention was to obtain data for the family 
(Chevny; pea 124s 


A common assumption among researchers is 
that respondents can readily give answers that 
may in fact require a considerable amount of 
work to complete or may require some time 
searching for bills or household records. For 
example, surveys among farmers may ask for too 
much detail on amounts of land planted with 
different crops, the crop yields, costs of 
seed, fertilizer, machinery, gasoline and so 
on. Other questions may expect far too good a 
memory from respondents. Demographic surveys 
have consistently found that women, especially 
older women, under-report the number of 
children they have ever borne, particularly 
tending to forget children who have "left home" 
or died. The 1954 Census of France asked women 
"Have you had children?" (If yes) "How many?" 
with an instruction to count all children 
including those who had died but excluding 
still-births. A special questionnaire to a 
sample of women asked in addition for the first 


name, sex, date of birth, and date of death of 
those who had died. The special questionnaire 
yielded a higher birth-rate, and analysis 


revealed that women who answered the standard 
questionnaire tended to omit children from an 
earlier marriage, and (from not reading the 
instructions) those who had died (Chevry, p. 
197) 


Respondents who have been admitted to a 
hospital for treatment of a disorder often 
cannot give a clear complete explanation of 
what was done to them, either because they were 
never -'told- by" the “hospital-"'or, Vdoctor,-*or 
because the explanation was too detailed or 
impossible to understand. Health survey-takers 
have found it better to work with hospital 
records than to rely on patients' answers. 


Another phenomenon that might occur is known 
as "forward telescoping" - the general tendency 
for people to think that an event occurred more 
recently than it did. In expenditure surveys 
asking for recall over, for example, 12 months, 
there is a tendency for respondents to report 
purchases that were in fact made longer ago. 
Some experimenting has been carried out with a 
technique known as "bounded recall" or "depen- 
dent interviewing". 


In this approach, shorter time-periods, such 
as three months, are used with data obtained 
from the same respondents every three months 
over a 12-month period. Interviewers are pro- 
vided with details of purchases reported by the 
respondent in the previous period (for example, 


by providing them with a copy of the question- 
naire they completed last time). Claimed 
purchase of, for example, an item of furniture 
in the current period can be checked against 
reported purchase of a similar item in _ the 
previous period. The respondent can be invited 
to confirm that. the current period item is 
indeed different from the previous period. 


Backward telescoping can also occur, but is 
much less common than forward telescoping 
(Sudman and Bradburn 1982, p. 18). 


Sensitive Questions 


Sensitive questions (that is, ones perceived 
as irritating or threatening), for example, 
questions on income or in some situations, age, 
tend to get low response rates and may trigger 
a refusal by the respondent to cooperate any 
further. They should not be placed at the 
beginning of a questionnaire. At this time, the 
designer is primarily concerned with gaining 
the respondent's trust and getting him inter- 
ested in even answering the questionnaire. 


They should be positioned in the question- 
naire where they are least likely to be sensi- 
tive, such as where the subject being discussed 
would suggest they might fit. Thus in a survey 
on work experience that contains a set of ques- 
tions on the current job, it would be logical 
to ask about the present hourly wage-rate, plus 
number of hours usually worked, at the same 
time as questions on when the- respondent 
started the job, what the job is, who he works 
for, and so on. In this way, the sensitive 
income-question should not seem out of place or 
threatening. Sometimes, sensitive questions are 
better at the end of the questionnaire. Any 
hostility or bias evoked by them will influence 
few or no further questions. Most of the infor- 
mation will already have been obtained. If 
there is evidence that responses to following 
questions have been severely affected, it may 
be possible to develop procedures to impute 
answers that can be judged more acceptable, or 
to allow for a "no-answer" class in place of a 
doubtful answer. 


Tests or past experience are desirable for 
determining first, whether any questions are 
perceived as being sensitive and secondly, the 
best location in the overall sequence for such 
questions. 


The questionnaire designer should also look 
out for questions that are perceived by 
interviewers as being sensitive when in fact 
they are not seen as sensitive by the majority 
of respondents. A few hostile reactions from 
respondents early in an assignment could lead 
an interviewer to this view, when in fact 
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subsequent respondents may not have’ strong 
feelings. An interviewer may be tempted to omit 
or reword such questions, putting them in a 
form that the interviewer thinks is likely to 
make them less sensitive. 


There are a number of other ways in which 
threatening questions may be made less sensi- 
tive: 

- open-ended questions using familiar words 
may be better than closed-ended, and long 
better than short, to get frequency informa- 
tion, or by asking for a narrative response. 
Do not strive for a very precise response; 


- try using words that suggest that many others 
behave in the sensitive way or have the 
characteristics that are seen as sensitive. 
Wording such as "did you happen to" may not 
helps; 


- put the threatening topic in the middle of a 
list of topics, some more threatening and 
some less, to reduce the perceived threat of 
the topic to the respondent. 


As well, questions can be asked at the end 
of the interview on how sensitive the respon- 
dent found the topic, with responses used in 
evaluation of the quality of response (Sudman 
and Bradburn 1982, p. 55). 


Statement of Alternative Answers 


The respondent's train of thought is led by 
the choices offered in a question. If a ques- 
tion contains a selection of answers from which 
the respondent is to choose, it is best to 
present all possible answers and leave no 
alternative implied. If for example only one of 
two possible answers is presented, respondents 
are more likely to aqree with the answer pre- 
sented than to formulate the alternative in 
their minds and to choose objectively between 
the two. The following example presents two 
wordings of a question and the corresponding 
results when asked of two matched groups. Note 
that the first example presents only one possi- 
bility and leaves the alternative, that com- 
panies could avoid layoffs, to the respondent 
to formulate. 


"Do you think most manufacturing companies that 
lay off workers during slack periods could 
arrange things to avoid layoffs and give steady 
work right through the year?" 


Results: 


63% said companies could avoid layoffs; 
22% said they couldn't; and 
15% had no opinion. 


The question was then reworded to state the 
implied alternative that layoffs are unavoid- 
able. 


"Do you think most manufacturing companies that 
lay off workers in slack periods could avoid 
layoffs and provide steady work right through 
the year, or do you think layoffs are unavoid- 
able? 


35% said companies could avoid layoffs; 
41% said layoffs are unavoidable; and 
24% expressed no choice. 


About two thirds of respondents agreed with 
the idea presented in the first question. 
However, when the alternative was explicitly 
presented, the proportion agreeing with the 
initial idea fell to about one-third. 


The problem of motivating the respondent to 
formulate alternatives is magnified when there 
is more than one alternative to be considered. 
The following example illustrates the distribu- 
tion of responses to two questions, the first 
one with limited alternatives presented in the 
question and the second one with many more 
alternatives presented. The two questions were 
asked of the same group at two points in time. 
Note that the complex nature of the question 
presenting the larger number of alternatives 
required a visual aid. The responses’ were 
listed on a card which was handed to the res- 
pondent when the question was asked. 


"Suppose they do set up a plan to provide 
workers with unemployment and health benefits 
through royalty payments, who should manage the 


fund: the companies, the government or the 
union?" 

Per 
Results: cent 
Government 33 
Union 18 
Company 18 
Company and union 7 
Company and government 1 
Union and government 1 
All three 6 
No opinion 16 
"Which of these should manage the _ fund?" 


(PRESENT CARD) 


Government 

Union 

Companies 

Companies and unions 

Companies and government 

Unions and government 
Government, unions and companies 


2g 


Per 
Results: cent 
Government 18 
Union 4 
Companies 13 
Companies and unions 18 
Companies and government 8 
Unions and government 5 
Government, unions and government 2) 
No opinion 13 


When the possibility of combinations was not 
offered, only 15% went beyond the question's 
limited choice and volunteered combinations. 
Later, when the combinations were explicit and 
presented visually, about one half of the 
respondents chose from among the combinations. 
It is possible that the shift reflects changes 
in opinion that occurred from one survey to the 
next. However, the researcher (S. Payne) felt 
"it seems more likely that the difference in 
results arises from the explicit mention of the 
combinations in the second survey...". 


Flow of Questions 


Questions should be arranged in an order 
which meets the objective of good flow, without 
awkward or illogical jumps. 


The principles outlined below supplement the 
principles for division of a survey subject 
into sub-topics, and ordering of the sub- 
topics, as discussed earlier. 


Once a general topic has been raised, all 
questions on that topic should come together 
before a second topic is started. Thus, in a 
question on work experience, one group of ques- 
tions could be about all the jobs the respon- 
dent has had since he entered the labour 
force. Then another group of questions could 
explore his present job, asking more details 
about it. There would be no further reference 
back to past jobs. 


Each section of questions should have an 
introduction explaining the general topic to 
the respondent: "First, let's talk about all 
the jobs you've had since you started to work. 
What was the first job you had?..." "Now, let's 
talk about your current job. When did you first 
start working at that job?...". 


Questions of family members in a reference 
period can best be ordered so that all details 
are obtained for a given individual, before 
details are obtained for a second person. 
Initial questions would determine a _ limited 
amount of information about the household 
(number of people, their age, sexes, and rela- 
tionship to the head of the household). It 
might then be logical to follow with questions 
on each member over a longer reference period. 


It can be confusing to respondents (and 
possibly to interviewers) to ask for a listing 
of events, go back to ask about a characteris- 
tic of each event in turn, then go back again 
to ask about another characteristic. The 
designer may be thinking of analysis of simple 
characteristics, i.e., for all events or wages, 
but the respondent is more likely to be think- 
ing about all the characteristics of one 
event. As much as possible it is the logic of 
the respondent which should determine how ques- 
tions are worded and laid out. 


The approach commonly used in obtaining 
household data in a household roster or docket, 
is first to list all members of the household 
by age, sex, and relationship to the head. Then 
in a second stage, obtain all information about 
each individual in turn (e.g., education level, 
employment status, marital status, place of 
birth, places lived in, etc.). In this situa- 
tion, it seems more logical to obtain a limited 
amount of information at initial questioning in 
order to ask questions more intelligently in 
the second round. This two-step procedure 
appears to work well for interviewers. 


A pretest of the questionnaire on typical 
respondents is recommended as a means for 
determining whether the flow of items is in 
fact perceived as reasonable by respondents. 


Not only should the general flow of ques- 
tions appear smooth to the respondent, the 
respondent should also be able to see the 


e ae Statistics Canada Statistique Canada 


CANADIAN HEALTH AND DISABILITY SURVEY — CHILDREN 


Interviewer: If Vision Trouble (circle 36) in item A17, 
on page 2, then... 


1) go to B5 


Otherwise 7O go to B15 


Vision Trouble 


B5. At the last interview, it was reported that... had vision 


trouble not corrected by an aid. 
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relationship of the questions to the stated 
objectives. Questions should relate to the 
overall purposes. Otherwise, suspicion may 
arise just as it may with irrelevant opening 
questions. Questions for which the only purpose 
in obtaining answers is that of being "nice to 
know", and which are only distantly related to 
the objectives, should be avoided. 


questions should usually 
and usually from 


Where applicable, 
follow a chronological flow, 
present to past although sometimes reverse 
order can be more suitable. Chronological 
lead-ins can be valuable memory aids for ques- 
tions on such topics as migration, or job 
history. For example, questions on job history 
could precede questions on formal schooling. 


A further recommendation regarding flow is 
that a long sequence of questions in exactly 
the same form should be avoided, or be broken 
up by a different type of question. Lists of 
questions in the same form can be boring or 
irritating, or can put the respondent into a 
particular response set where he tends to 
answer mechanically without thinking carefully 
about the individual questions. 


Filter Questions and "Skip" Instructions 


A filter question can be used to exclude a 
respondent from a question sequence which does 
not apply to him because of his own particular 
characteristics or circumstances. For example: 


CONFIDENTIAL 
‘ when completed 1 


“Authority Statistics Act, Statutes of 
Canada, 1970-71-72, Chapter 15,” 


Interviewer: If Hearing Trouble (circle 34) in item Al wes 
on page 2, then... 


"O go toB16 
Otherwise 7O go to B27 
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Sometimes it is necessary to include an is directed to check the answers to two or 
"interviewer check" question, whose purpose is three questions, then follow the appropriate 
to act as a guide to more complex branching branching instructions. 


through subsequent questions. The interviewer 


bE od Statistics Canada Statistique Canada 


CONFIDENTIAL 
when completed 


CANADIAN HEALTH AND DISABILITY SURVEY — CHILDREN 


“Authority Statistics Act, Statutes of 
Canada, 1970-71-72, Chapter 15.” 


9. INTERVIEWER CHECK ITEM 
1 - yey es” in screening column of SECTION A ; 6 ee 
and age on the label is greater than or equal to 05 


“Otherwise 72 END 


Make an appointment to interview this respondent as per instruc- 
tions in your Interviewer’s Manual. 


At times, a conditional question will be translators are applied consistently throughout 
used to establish the relevance of subsequent the documents and that the technical termino- 
questioning, for example, "If you are working logy is as simple and as direct as possible. In 
or looking for work, what is your main reason?" all cases, literary niceties will be secondary 
as well as non-relevance, e.g., those who are to the principal aim of communicating meanings 
neither working nor looking for work may ignore clearly and consistently to respondents’ and 
the question. The main disadvantage with the interviewers and eventually to users of the 
use of a conditional question is that a blank results. One technique for improving’ the 
answer can be interpreted as a non-response to quality of translation is back-translation: the 
the question. It would be impossible to distin- original is translated, then retranslated back 
guish between a non-response to the question by another translator. The two versions are 
and a "not applicable" answer unless’ that compared to ensure that meanings have remained 
answer category is also provided. In this case, consistent throughout. 

a filter question such as "Are you now working 

or looking for work?" is preferable with res- Since translation is time consuming = and 
pondents answering "Yes" being directed to the affects all related documents, it is advisable 
question on the main reason for working or to test for and eliminate translation biases 
looking for work. If the response is "No", the during the design phase with small back-trans- 
respondent can be instructed to skip to the lation tests. Alternatively, use of two or more 
next appropriate question. translations in the pilot test can reveal the 


effects of differences. 


Translation Considerations A final opportunity exists for correcting an 

inadequate translation. Reports on survey field 

In surveys sponsored by the federal govern- operations can include comments on difficulties 

ment, documents are required in two languages. caused by inadequate translation. This intelli- 

Translated versions of the questionnaire and gence can be used to correct survey documents 
the interviewer's guide and training manual before the next round of interviews. 


will be needed. 
Once field work is underway, the success of 
the survey rests on the shoulders of the inter- 


In preparing translations, keep in mind that viewers. Interviewers must be carefully trained 
survey documents are technical instruments. As in the concepts and wordings and have at hand a 
such, they should convey a consistent message complete set of reference documents including 
or idea. Thus, the questionnaire designer will translated versions of the questionnaire or 


want to ensure that the terms used by the schedule. 


Parts of the Questionnaire 


A well-structured questionnaire will help in 
meeting survey objectives by improving the 
quality and quantity of response. The design of 
the questionnaire must satisfy the respondent, 
the interviewer, and the questionnaire design- 
er. 


With respect to the respondent, the ques- 
tionnaire should be designed to provide and 
maintain motivation to respond. Even for well- 
motivated respondents, interest may diminish: 


- if the questionnaire 
topic irrationally; 


skips from topic to 


- if many questions seem irrelevant; 


- if the interviewer gets lost in a complica- 
ted, unclear questioning sequence. 


Questionnaire design should overcome’ any 
Suspicions or doubts that the respondent may 
have concerning the legitimacy of the survey. 
Careful and effective sequencing may facilitate 
the respondent's ability to .recall specific 
items. For the interviewer's benefit, questions 
should flow in a clear and orderly manner to 
make administration of the questionnaire as 
easy aS possible. Precise instructions should 
be included on how to move ahead, how and when 
to skip to certain questions, and so on. 


In arranging the sections of a questionnaire 
into a sequence or order, a number of factors 
should be considered: 


- There may be a substantive need to ask a 
certain group of questions before others. For 
example, questions on awareness of an item 
(such as, a program, a service, or a proce- 
dure) should precede questions on the extent 
of knowledge or of use of the item. 


- Logic (particularly from the point of view of 
respondents) may suggest that some orderings 
of topics are better than others. 


- Sensitive topics should not come before a 
good rapport has been reached between the 
respondent and the interviewer. 


- Questions relating to common reference 
periods should be grouped together. The most 
recent reference period coming first with a 
systematic lengthening of the reference 
periods. 


- The most important topics should be included 
first followed by sub-topics which could be 
cut back or cut out entirely in the interests 
of respondent fatigue, quality or bias. 
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- Some questions or sets of questions must be 
asked before others as they serve as screens 
or filters to route respondents to different 
sets of questions. 


- Topics ought to be grouped into those that 
can be expressed in questions that will be 
answered adequately by proxy, and those that 
can only be answered by direct response. 


- Questions on use should not precede questions 
on general knowledge. 


In self-enumeration surveys, where the res- 
pondent is free to read the entire question- 
naire before answering any questions and may go 
back and change his response to earlier ques- 
tions, awareness cannot reliably be measured. 
Questions on use or experience will often say 
enough about the degree of awareness that the 
latter need not be asked. 


Introduction 


For most surveys, respondents are under no 
legal obligation to respond. Even where res- 
ponse is mandatory, the law is rarely invoked 
to force response. 


Consequently, other means must be found to 
arouse the respondent's interest. He must he 
motivated to think carefully about the answers 
to the set of questions asked so that high 
quality data are assured. An explanation of the 
purpose of the survey and a request for cooper- 
ation should be a part of the first information 
provided to him. This can be done through an 
introductory statement. In interviewer-adminis- 


tered surveys, the interviewer can make the 
statement, immediately after he introduces 
himself. The statement can be printed on the 


first page of the questionnaire or be included 
in the interviewer's manual or an introductory 
letter can be mailed in advance to. the respon- 
dent. The interviewer may also carry a copy of 
the letter with him to show to respondents. 


The statement of purpose should demonstrate 
to the respondent how survey results are likely 
to be of benefit to him. Other elements of the 
introductory statement can be: 


- an assurance of confidentiality but only if 
such assurances can in fact be made. Confi- 
dentiality assurance is especially important 
when the survey contains questions that an 
individual perceives as personal, or requests 
information which a business establishment 
would not want a competitor to obtain; 


- an identification of who is responsible for 
the survey, and who is carrying it out; 


eit 


- a statement for the legal authority to carry questions about the survey. Providing this 
out the survey, if there is one; kind of assurance often leads to greater 
willingness to respond, even if the respon- 

dent never contacts the senior official. An 


- the name of a manager or other person who example of an introductory statement is given 
could be contacted if the respondent has any below: 
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Special Surveys Co-ordination Division 


1976 SURVEY OF FITNESS, PHYSICAL RECREATION & 
SPORT 


October 1976 
Dear Respondent, 


The purpose of this survey is to provide comprehensive data on Canadian participation in fitness, 
physical recreation and sport activities. It is being conducted for the Fitness and Amateur Sport 
Branch of Health and Welfare Canada. Data from the survey will be analysed by the Fitness and 
Amateur Sport Branch to aid in the planning and development of new programs, and in the 
investigation of exercise, physical recreation and sport participation of Canadians. 


The Statistics Canada interviewer has left this questionnaire for you to complete and will call back 
to pick it up on the date shown below. 


We would like you to take a few minutes to answer the questions on the next few pages. Answers 
to most questions in this survey require only a check mark in the appropriate box [VJ or circle ). 
If you should have any difficulties with a question, note this in the space on the back of the form 
and ask your Statistics Canada interviewer for assistance when he/she returns. I assure you that 
the information given by you is confidential and will be used for statistical purposes only. 


Thank you for your co-operation. 


Peter G. Kirkham, 


/ ee 


Chief Statistician of Canada. 


This survey is conducted by Statistics Canada on behalf of the Fitness and Amateur Sport Branch, Health and Welfare 
Canada, under authority of the Statistics Act, Chapter 15, Statutes of Canada 1970-71-72. 


Administrative Data - geographical codes (identifying the province, 

city, district or other class) and a serial 

Administrative data, usually recorded near numbering code for interviews within a 
the beginning of the questionnaire, include: district; 


- identification codes for the survey ("Survey 
No. 1982-04" or "Form 82-H1"), and for the - dates of interview and follow ups; _= and 
interviewer assignment; starting and ending times; 
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- codes to indicate the extent or quality of 
response, or a reason for non-response; 


- codes to identify who gave the _ response 
(e.g., the specific respondent intended, or a 
proxy such as another member of the family or 
household) ; 


- codes for field editing (i.e., to verify that 
the field supervisor has checked the ques- 
tionnaire and answers are where they should 
be, and legible and complete). A separate 
code may be useful for field coding (i.e., to 
verify that the interviewer or supervisor has 
coded questions that need coding in_ the 
field); 


- name and address of the respondent, parti- 
cularly if the design calls for response from 
"the head of the household" whose name may 
not be known in advance, and must be obtained 
at the beginning of the interview. 


Placing control or identification informa- 
tion at the top of the first page often makes 
it easier to find a specific questionnaire 
later if it has to be retrieved from storage, 
as may happen during office editing or capture 
of the data for machine processing. 


The preceding example is taken from the 
Canadian Health Survey of 1978/79. The upper 
part of the Household Record Card contains, to 
the left, spaces for sample identification 
codes and for the household address. The upper 
right corner provides space for interviewers to 
record dates and times of contact with the 
household, and for a code for response. The 
shaded area in the lower part of the question- 
naire (marked "CONTROL") was used to record 
completion of stages of the survey (several 
questionnaire forms were used). 


Main Body 


After the introductory parts of the ques- 
tionnaire including administrative data, comes 
the main body of the questionnaire which con- 
tains the substantive questions on the subjects 
of the survey. As discussed, these may be sub- 
divided according to logical breakdowns or 
groupings of sub-topics. 


The opening questions of the main part of 
the survey must gain the respondent's attention 
and interest. Generally, the opening questions 
should put the respondent at ease. Difficult or 
especially threatening questions in the early 
stages may lead to inaccurate response or non- 
response and should be avoided. 


In some surveys, the opening questions may 
serve to establish that the respondent is a 
member of the survey population. For example, 


ey) 


in a survey of employment, the initial contact 
with a responsible adult could be to list all 
those persons living permanently in the house- 
hold who have a job. The next question ad- 
dressed to that person could serve to confirm 
that he is a member of the household and quali- 
fied to respond. In the affirmative, the third 
question could be to ascertain if he is 
"employed". 


The opening questions also serve as a learn- 
ing process, educating the respondent on his 
expected role concerning the topics raised, the 
detail required in answers and how to answer 
questions. This further emphasizes that the 
initial questions should be easy to answer. The 
respondent is required not only to _ become 
attuned to the topic introduced on the ques- 
tionnaire but also to learn what is required of 
him as a respondent. 


Classificatory Data 


Questions to be used in classification and 
analysis are usually grouped together. Often 
these are demographic descriptors (e.g., age, 
sex, education level, occupation, income, and 
so on) as illustrated in the example below 
drawn from the 1976 Survey of Fitness, Physical 
Recreation and Sport. 


Sampling or respondent-selection procedures 
may require some of these questions to be asked 
early in the survey (e.g., in order to pick a 
respondent according to certain characteristics 
such as physical variables or position in the 
household). Most commonly the classificatory 
questions are together at the end of the 
questionnaire because they can be more readily 
justified to respondents at that point as 
wanted just for analytical purposes. 


It is generally argued that only’ those 
classificatory questions that are expected to 
be used for specific analyses should be asked. 
But questionnaire designers in government agen- 
cies often cannot foresee all the analytical 
uses that are likely to be made of their survey 
data. At some future time somebody somewhere 
may wish to make comparisons by age or marital 
status with other survey results, or with 
administrative data. To satisfy these future 
researchers, all the usual demographic and 
classificatory questions should be asked. 


Since some classificatory questions are 
asked of respondents in almost all household 
surveys, it is useful to consider the benefits 
of aiming for long-term comparability among 
surveys. All too often, an analysis of two or 
more surveys, seeking differences or trends, is 
flawed because it is not possible to group 
respondents in both surveys according to the 
same demographic classifications. For example, 
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1976 SURVEY OF FITNESS, PHYSICAL RECREATION & 
SPORT 


This section collects information 
which is useful in grouping 
your replies with other Canadians 


in this survey. 


Were you born in Canada? 
804 805 F x 
[ ] Yes No — In which country were you born? 


817 aan 
Great Britain 


In which province or territory 
of Canada were you born? C] 818 


ee Newfoundland [] i Ontario 

Bg Nova Scotia [ey Manitoba 

[J we New Brunswick Ge Saskatchewan 
lias Prince Edward Island ae Alberta 

[]°° Quebec [_]° British Columbia 


Ea 86 NWT, Yukon Print country of birth 


What language do you most often speak When you were 13 years old what was the population 
at home? of the city, town, village, municipality in which 


United States 


| aM Germany 
[IP italy 
ei France 


822 
(ee Other/(specify) 


you were living? 


2 828 831 
[]® English [| italian 100,000 or more []~ 1,000 to 14,999 
829 832 
[]** French [_ ]®” other 30,000 to 99,999 [_]® tess than 1,000 
(specify) 


830 833 
[]” German [_] 15,000 to 29,999 []® don't know 


How much do you weigh? 


How tall are you? 837 «838839 check only one. 
OR i 
Feet Inches Centimetres 90 and less 91-100 101-110 111-120 121-130 
Ibs. Ibs. Ibs. Ibs. Ibs. 
To what extent has your weight changed 840 84) 842 343 844 
or fluctuated over the past 12 months? CL] L] a L] L] 
8: 8 kp kg. kg 
Check only one 40.9 and less} 41.4-45.5 | 45.9-50.0 50.4-54.5 55.0-59.1 
855 ; 131-140 141-150 151-160 161-170 171-180 
re Remained steady to ine Ibs. Ibs. Ibs. Ibs. 
within 5 pounds/ 845 846 ei fe ks ou 
2.3 kilograms a CJ kg. kg. C 
fe 59.5-63.6 64.1-68.2 68.6-72.7 73.2-77.3 77.7-81.8 
[J Fluctuated 6 to 
10 pounds/ 2.7 to re) > 
4.5 kilograms 181-190 191-200 201-210 211-220 ver 220 
ibs: Ibs. Ibs. Ibs. Ibs. 
850 851 852 853 854 
|” Fluctuated more L] L LI LI LI} 
Bae i kg kg kg. kg. kg 
t an pounds/ 82.3-86.6 86.8-90.9 91.4-95.4 95.9-100.0 Over 100 
4.5 kilograms JE ae J 
In the last 12 months, has your doctor told you that you suffer from a heart related ailment? 
858 859 860 
[J Have not seen a doctor No Yes 
What was your total personal income How many automobiles are available for use by 
before taxes and deductions in the last members of your household? 
12 months from all sources? (include 


earnings from employm q i 
gs fi ployment, pensions, family 872 


« ne 870 87] 
allowances, interest, dividends, rents) fee None One [] Two Pay Three or more 
(Please check one box.) 


861 865 873 876 
N | a 
LJ One $10,000 - 14,999 j office use only ig bbe | 
862 866 
[]™ Less than [| sis,000-19,999 


$4,000 THANK YOU FOR PARTICIPATING IN THIS SURVEY! 
Fas $4,000 te We would appreciate any comments which you wish to make on government 
,000-6,999 $20,000-29,999 programs related to fitness and amateur sport. Please use the next Page to 
do so 
864 
[]™ s7.000-9,999 — [_]™ $30,000 or over 


41402-13-1: 13-08-76 


ages of respondents may have been obtained 
according to precoded age-ranges, with the 
ranges different between the studies; or the 
geographical classifications were different so 
thatupake majommMeityy turns®yout “toy cinelude tra 
neighbouring or suburban area in one study, but 
exclude it in the other. 


Statistics Canada's "Social Concepts Direc- 
tory" and Sudman and Bradburn (1982), Chapter 
7, are two sources among many of standard 
classifications, particularly of demographic 
data. Others include: classifications of occu- 
pations (from the International Labour Organi- 
zation), of education (UNESCO), diseases (World 
Health Organization), and the UN Statistical 
Office reports. These standards have been care- 
fully developed, so a survey designer should 
consider their usefulness before repeating any 
of this work. 


Geographical information, frequently wanted 
for classification purposes, can raise some 
problems. Identification of regions or dis- 
tricts can be difficult if there have been name 
changes. Changes of names or, even more, 
changes of boundaries, can lead to complica- 
tions in getting correct answers (e.g., in 
place-of-birth questions). In general, it is 
preferable to use names with which the majority 
of respondents are familiar, even if they may 
mean the use of a name that is no longer offi- 
czally ‘correct: 


Interviewers should be advised, particularly 
when interviewing older or less-well-educated 
respondents, to check whether they are refer- 
ring to a district or region by its old or new 
name or boundaries. Place names may be ambi- 
guous in the areas they cover, or may refer to 
a part (usually the older or core part) of a 
city, as well as a larger urban area _ that 
includes the core part. Respondents may have to 
be asked if they mean the original core area or 
tig Slarger “tcity™ “areay?%e.g?, “Téronto. or 
Etobicoke. 


Open-ended vs. Closed-ended Questions 


Questions may be classified into’ the 
following recognizable type: factual versus 
opinion, and narrative versus numeric. However 
in dicussion of advantages and disadvantages, 
the most important and useful distinction to be 
made in the classification hierarchy is that of 
"open-ended" versus "closed-ended" questions. 


The distinction between open-ended = and 
closed-ended questions concerns the degree of 
freedom given the respondent in choosing his 
answer. An open-ended question allows the res- 
pondent to create a response on his own, and in 
his own words or numbers; a closed-ended ques- 
tion restricts him to choosing among alterna- 
tive answers specified in the questionnaire. 


The following is an example of a question 
whose content can be adapted to either type: 


Open response or open-ended question: 


Q. 22 Crime Incident Report 1984 


22 How were you threatened? Any other way? 
(Mark all that apply) 


iC) Verbal threat of rape 
A@) Verbal threat of attack (other than rape) 
sC) Weapon present or threatened with weapon 


K@) Attempted attack with weapon e.g. shot at 


Go to 31 


s_) Object thrown at person 


&C) Other (specify) 


Q. 82 Crime Incident Report 1984 


Closed response or closed-ended question: 


82 | am now going to describe different circumstances that may 
affect whether or not an incident is reported to the police. Did any 
of the following have anything to do with why this incident was 
not reported to the police? 

Yes No Don't 
know 


— 


. Nothing was taken or the items 
ol 02 03 
were recovered .............. © © O 


N 


. The police could not do anything ,, 05 98 
AD OUR cot pi fc ata nL oe O © 


Ww 


. There was fear of revenge by the ,, 08 #8 
Offender) . .cA2% «Gas Valse O © ) 


4. Did not want to get the offender in i re 
trouble with the police......... u® © e) 


The incident was too minor or it 
: 13 14 15 
was not important enough....... © O O 


oa 


6. It was too inconvenient or did not 


want to take the time.......... oe) sk od dw) 


. The incident was a personal matter 
: ? 19 20 2 
and did not concern the police... . @ O O 


8. The incident was reported to another 
: 22 23 24 
official, such as a security guard .. . © © O 


9. Concern with the attitude of the 


police or courts towards this type = oe Pr 
Of incident hak «oi sue. nee es (@) © © 


NS 


A third category is made up of those nomi- 
nally open-ended questions that have an impli- 
cit finite number of choices. Examples are 
questions such as: 


"What was your age at last birthday?", "How 
many people were living permanently at this 
address on June 3, 1981?" These examples would 
become closed-ended questions if lists of 
acceptable answers or answer classes are pro- 
vided. 


For example, a respondent could be asked to 
provide his age in 5-year or 10-year age ranges 
by asking for the range in the question: 


"In which of these age-ranges did your age 
at last birthday fall?" (INTERVIEWER: READ 
LIST) 


{_] under 20 
[__] 20-24 
ey 25=20 [Sy 30=33 
[305545 etco. [_] 40-49, etc. 


A variation of the types described above is 
sometimes called a partially closed-ended or a 
semi-open-ended question. To the respondent, 
the question sounds open. He is simply asked, 
for example, "What have you done to look for 
work during the last four weeks?". But the 
interviewer has a precoded sheet for the res- 
ponses. As the respondent talks, the inter- 
viewer matches the comments against the list of 
acceptable responses and codes them. The time 
required by interviewers to write out verbatim 
responses is eliminated. However, the question- 
naire designer must predict the likely range of 
responses to the question, not always) a 
straightforward task. Pretesting can be help- 
ful to check whether the major choices have 
been included. Because..of. the diffaeultay of 
predicting all the possible answers and incor- 
porating them into precoded answers, such 
questions often finish with a category "other 
(SPECIFY)", providing space for the interviewer 
to write in an answer that does not fit one of 
the listed categories. 


waa under 20 
(me 2029 


or 


The "Other" answer category can also be use- 
ful for an interviewer who cannot immediately 
see which of the listed categories is suitable 
for the answer. After the interview, he can 
re-code the answer into the correct category. 
It may also become apparent that a class or 
concept not precoded receives enough mentions 
to merit separate analysis. 


25. WHO PAID THE FEE OR TUITION FOR THE COURSE? 
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Advantages and Disadvantages 


Open-ended questions calling for numerical 
responses, or responses that are clearly limi- 
ted to a small number of categories, require 
that a quantified answer be given, that the 
amount of space on the questionnaire accommo- 
date the largest likely answer and that the 
size of the field in the computer record layout 
be large enough to capture that largest likely 
answer. 


It might be assumed that a question calling 
for a numerical answer is likely to be under- 
stood the same way by all respondents, but this 
may not be the case. Open-ended numerical 
response questions such as "How many cattle do 
you have?" leave the respondent free to name 
any number, whether exact or rounded off or to 
come up with answers such as "40 or 50", "a 
lot", "not enough", or "not as many as last 
year". The questionnaire designer must accept 
them and provide appropriate answer _ spaces. 
Alternatively, he can add supplementary ques- 
tions designed to change the respondent's 
understanding of the question to yield the 
required single numerical answer, or _ train 
interviewers to probe for a more _ precise 
answer. Failing this the questionnaire designer 
must attempt to utilize the closed-ended format 
in an effort to achieve consistency and reduce 
questioning. 


In preliminary exploratory studies, open- 
ended questions are useful in generating ideas 
on how respondents view the world. Words and 
phrases used by respondents can be gleaned for 
use in questionnaire development. Contributions 
from several respondents may also help deter- 
mine a reasonable range of responses for a 
structured multiple-choice question. The ques- 
tionnaire designer may obtain a more objective 
view and word his questions more appropriately 
after an exploratory study using open-ended 
questioning techniques. 


Open-ended questions can be used in a number 
of ways in main survey questionnaires. They 
imitate the every day process of human communi- 
cation. They may be used to warm up a respon- 
dent at the beginning of an interview, to 
introduce ;him-;to ithe,..topic, -to.-pub hime 
ease. They give the respondent an opportunity 
for self-expression which acts as an incentive 
for continued participation. 


Open-ended questions are advisable when it 
is expected that some respondents may have no 
information about an issue, where structured 
items might induce a false response, any 
response being better than no response at all 
for fear of appearing ignorant. 


Requests for further explanation, more 
detail, and general probing for a more complete 
answer, usually take the form of a series of 
open-ended questions.... "Can you tell me more 
about that?, "Why do you say that?" etc.,.... 
After a staccato series of closed-ended ques- 
tions, open-ended questions may lend a pleasant 
change of pace. 


Open-ended questions can be useful as a 
transition between two different topic areas in 
a questionnaire. 


Open-ended questions that call for a narra- 
tive response, such as on knowledge, experi- 
ence, attitudes or beliefs, have a number of 
limitations. Their disadvantages become appa- 
rent when attempts are made to code the data 
for statistical analysis: 


- first there is a need to develop a coding 
frame, and a procedure and organization for 
coding. Precoding which is possible with 
closed questions, as well as eliminating this 


work, cuts down on office-coding error, 
although interviewer-coding error remains 
unknown. 

- secondly, the coding process makes implicit 


assumptions that the results of the categori- 
zations will yield data that can be analyzed 
statistically as though they came from 
closed-ended questions, an assumption that 
may or may not be valid. Reticent people are 
often less detailed in their comments though 
equally aware and knowledgeable in terms of 
experience. Variations in respondent verbo- 
Sity suggest that the final questionnaires 
will capture unequal proportions of each 
respondent's experience, particularly where 
the number of different answers is left to 
the discretion of the respondent as is the 
case in open-ended questions. 


With the close-ended approach the designer 
can present a complete list of possible answers 
or of answer categories to respondents, and be 
sure that all respondents are answering the 
question in the same frame of reference. He can 
also be sure that all respondents have had an 
equal opportunity to select their answer(s) 
from the same range of answer choices. If it is 
difficult or impractical to present a complete 
list of answer choices, the list as presented 
can end with "other (SPECIFY)" to give respon- 
dents and interviewers a place to record this 


answer. However, no more than a small propor- 
tion of all answers should fall in the "other 
(SPECIFY)" class; otherwise, there will be an 


extensive need for the work of 


analysing. 


coding and 


by 


-to be read out. 


Commonly a long list is presented in written 
form to respondents. In telephone surveys where 
this is not possible, the item-list will have 
Difficulties in retaining the 
respondent's attention can arise if the list is 
too long. 


Calling for a quantitive answer in a pre- 
coded range can serve to make sensitive ques- 
tions less sensitive. For example, asking for 
exact income figures is frequently met with a 
refusal and sometimes with a complete termina- 
tion of the interview. It may be adequate for 
analysis to accept the range within which the 
respondent's income falls. Respondents can even 
be shown a prompt card on which income ranges 
are identified by a letter, for example: 


A - Under $5,000 a year; 
B - $5,000-$9,999 a year; 
C - $10,000-$14,999 a year; 


and be asked to name the letter identifying the 
income-class in which their income falls. This 
technique is also useful with questions calling 
for a narrative answer, about knowledge or 
experiences, that may tend to be sensitive to 
some respondents (e.g., health, family plann- 
ing). Presentation of a list suitably worded 
may encourage a respondent to indicate a 
correct answer that he would have hesitated to 
express in his own words in reply to an open- 
ended question. 


Open-ended questions tend to be less satis- 
factory in  self-enumeration questionnaires. 
They call for some effort on the part of 
respondents to write out their answers, a more 
difficult job than checking off answers from a 
list. There can also be no control over the 
words used by respondents, or whether’ the 
answer is even relevant to the question. Often 
the answers are incomplete or vague. At times, 
too, skill may be needed to interpret poor 
handwriting. The result can be less complete 
answers than for closed-ended questions. 


The main disadvantages of "closed-ended" 
questions are that they assume that the ques- 
tionnaire designer is able to identify the most 
commonly made responses and to design catego- 
ries for them. Closed questions also tend to 
"lead" the respondent. 


taken in drawing up closed- 
ended questions to provide categories that 
reflect the respondent's characteristics or 
experiences. For questions asking for quantita- 


Care must be 


tive answers, the classes’ should reflect 
"reasonable" distributions depending on the 
objectives of the question, the expected 


analyses to be made of results and on prior 
knowledge from other sources of the likely dis- 
tribution of the variable in the question. 


Using income classes as an example, it may 
be reasonable to ask for income in five or six 
classes, aiming to arrive at roughly one fifth 
or one sixth of the respondents per class. But 
if an analysis is planned of characteristics of 
respondents at subnational levels, the use of 
income classes that reflect national averages 
may lead to grossly skewed distributions where 
there is regional disparity, and consequent 
loss of analytic power. In such a case consi- 
derably more than five income classes may be 
necessary with the option to collapse classes 
differently for each region. 


Sometimes prior studies are available where 
open-ended questions have been asked, to deve- 
lop a pattern of classes of answers. In using 
these sources, care must be taken to use words 
for the items that were used by respondents. 
Technical descriptions given by the question- 
naire designer or analyst, possibly not mean- 
ingful to respondents, should be avoided. Where 
there are no prior studies, it is very desira- 
ble to carry out a small-scale exploratory 
study using open-ended questions, just in order 
to have a source of answers from which a 
closed-ended list can be developed. 


Presentation of a list of answers to a 
question asking about knowledge or experiences 
may turn the question into a leading question, 
suggesting answers to the respondent that he or 
she may not otherwise have thought of, or had 
considered insignificant or irrelevant. Respon- 
dents will often wish to appear knowledgeable 
or intelligent and pick answers that make them 


Limited-choice Question 


22. BEFORE YOU STARTED THIS TRIP DID YOU TRY TO OBTAIN ANY INFORMATION ABOUT THE DESTINATION? 


Yes 1 


No al®, Go to 25 
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appear to have had wider experiences or know- 
ledge than is true. 


Varieties of Closed-ended Questions 

The three simplest variations of closed- 
response questions are the "limited-choice", 
"multiple-choice" and "checklist". The most 
common version of the "limited-choice question" 
is any that expects a "yes" or a "no" as an 
answer, such as "Did you smoke any cigarettes 
yesterday?" 


The "multiple-choice" question asks the 
respondent to select one and only one answer 
from a list of three or more categories or 
classes. The "checklist" question" invites the 
respondent to pick one or more answers from a 
list of two or more items where, by the nature 
of the question and the subject-matter, more 
than one answer is logically possible. To make 
this clear to respondents, checklist questions 
are often accompanied by an instruction "Choose 
one or more" or "Pick as many as apply". The 
interviewer must pause after each item to 
obtain a "yes" or "no" answer. Sometimes, it is 
not immediately apparent whether the question 
is a multiple-choice or a checklist; instruc- 
tions to interviewers or respondents ought to 
make clear which it is. An open-ended "anything 
else?" may usefully be added. 


The three kinds are illustrated below, in 


three sequential questions taken from a 1983 
Tourism Attitude and Motivation Study. 


ay 


Multiple-choice Question 


DuningythenthiOmaw iia oneeeee arene Le) 
Same day as departure ......... 2.0 
(akorsxel Wie Sythe S Sueenen yo o-eeaes PAS) 
A*toiG: dayst = aeeee ee Dan oer ee oF @ 
Wee Kiger... Sipe eee cree en. a AG) 
ZEWEEK Sin on ee Se a ea ere nes) 
SIWEEK Stresses Recah a meena aera ee) 


21. HOW LONG BEFORE YOU STARTED THIS TRIP DID YOU DECIDE ON THE DESTINATION? 


ANN COCKS Sta nee te aa ee eater On) 
Datom/awWeeks. nantes met. cate en) 
Prikoy SHANKOMNINSe gene tea o-8 Sock eee ee 
4 toroimonthss se. 24 aoe i= en 14© 
ZetOnmlel MOUS ay pees) ecu eeene OQ 
Zr olenmOnesimO Mth Smemserus arena We) 


Checklist Question 


Home Of friends/relativeseame. o.chee tt’. . ne 
Hotel/Motel/Resort Lodge ............ “6 
Tourist home/Guest house ............ 1 @) 
Commercial cottage/Cabin ............ 4© 


A further type is the ranking question, in 
which respondents must arrange all of the given 
alternatives in an order corresponding to some 
specified criterion. Here is an example: 


"dere is a list of reasons you might have for 
buying your foodstuffs at the store you usually 
buy them from. Please rank them in order of 
importance for you. Put a (1) beside the reason 
that is most important to you for using the 
store, then a (2) beside the reason that is the 
second most important reason, and so on." 


[__] Convenient, close to home or workplace 


[__] Low prices 
yt Good quality 


[| Possible to buy on credit 
[_] Good range of choice 


[__] Convenient hours of business 


Cautions 
The Need for Exhaustive and Exclusive Lists 


The validity of the data from closed-ended 
questions rests on an assumption that the 
alternatives or choices offered in the question 
are mutually exclusive and exhaustive. Data 
from the following question, for example, could 
not be meaningfully analysed: 


"What is your age in years?" 
[__] 20-30 

{| 30-40 

[_] 40-50 

{aeons | 50,.0remore 


The categories are not mutually exclusive. 
In an interviewer-administered questionnaire, 
the interviewer, receiving an answer "30" has 
two places to record the number, and no guid- 
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29. WHAT TYPE(S) OF ACCOMMODATION DID YOU USE ON THIS TRIP? (MARK ALL THAT APPLY.) 


Campground) Mrailempank a) ene ene 1) 
Cruisesshitpec taunt oan ccc eis eee eee &@) 
Privatercottage/Cabimizsdkeeot ca 214s meee 1© 
Other (specify) iD) 


ance on which one to choose. It is impossible 
to judge what would be the likely pattern of 
choice of interviewers in distributing the 
answer between the "20-30" and "30-40" answer 
categories. The same applies to self-enumerated 
questionnaires. 


The categories are also not’ exhaustive. 
Where is the interviewer, or respondent, to 
record the answer if the respondent is aged 18? 
list at 


One way in which the categories 


could be improved is: 


{_] under 20 

[ faafie20=29 
[| 30-39 
[___] 40-49 
[___] 50 or more 


(ary not stated, refused 


[| not known. 


Sometimes it is not easy to be sure that 
list categories are mutually exclusive. This is 
particularly the case for questions on know- 
ledge or experience. Testing and other develop- 
mental work may be required to arrive at items 
that are perceived as mutually exclusive by 
most respondents. Exhaustiveness may need to be 
accounted for by use of a final "other" cate- 


gory. 


Multiple-choice Questions 


Sometimes, for multiple-choice 
collapsed frequency counts are employed in 
analysis. A respondent could be asked if he 
takes part in some activity, or has some expe- 
rience, and if so with what regularity; "very 
often, quite often, not particularly often, or 
not very often at all". Subsequent analysis 
could be such that the first two and last two 
classes are combined or collapsed to reduce the 
classes to two: "often" and "not often". 


questions, 


If the data are going to be collapsed any- 
way, the reader may ask, why not ask a two- 
choice question in the first place? Shouldn't 
one use the simplest question formatting that 
will test the hypothesis of interest? 


In general, it is desirable to make ques- 
tions as simple as possible. However, there is 
always the possibility that the planned com- 
bining of answer classes is found to _ be 
unsatisfactory or even incorrect for analysis, 
once the pattern of answers to the broader- 
choice question is examined. As well respond- 
ents can sometimes be discouraged by not being 
offered what they see as a wide enough choice 
of answers. They prefer to be able to indicate 
greater degrees of difference in behaviour or 
experience than a simple two-choice answer 
would provide. Thus, in order to keep respond- 
ent interest in the survey at a high level, it 
may be desirable to offer more than two 
answer-classes, regardless of anticipated ana- 
lysis of results. 


Checklist Questions 


A checklist question is appropriate when 
there is more than one alternative which may be 
"correct" for a given respondent. He is asked, 
by a series of yes-no questions, to report all 
the alternatives which apply to him. 


Most statistical analyses of checklist data 
presume: 


- that the alternatives are distinct, that is, 
that the items are exclusive. Otherwise, a 
respondent may use two categories to reflect 
the same response, and that response will be 
"double-counted"; 


- that the alternatives are independent. The 
checking-off of one alternative should have 
no effect on whether or not some particular 
other alternative is chosen; 


- that all respondents perceive the same con- 
straints in the number of allowable alterna- 
tives. Respondents should be advised to pick 
as many as they want. Interviewers could be 
instructed to ask "any others?" after each 
choice until the respondent ceases choosing. 


Ranking Questions 


In deciding whether or not to use ranking 
questions, some consideration must be made of 
the special analytical problems they present. 
Not only are there several ways to analyse 
ranking data, but different ways may give dif- 
ferent results. a/Thus) “wt vers. Sparticularly, 
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important to be precise concerning the research 
question the analysis is intended to answer. 
Two possible indices for analysis are: 


- frequency counts of favoured alternative or 
most frequent experience. For what percentage 
of people is alternative X the favourite or 
most frequent, i.e., chosen as first-choice 
in the rankings? How does alternative X 
compare with others in this respect? 


- favourability judged by taking all rankings 
into account: What is the average rank that 
alternative X obtained? How does alternative 
X compare with others in this respect? 


Averaging of ranks relies on an assumption, 
that the ranks are on an "interval scale", 
i.e., that the interval between rank 1 and 2 is 
perceived to be the same distance as the inter- 


val between 2 and 3, 3 and 4 and so on. Only 
this equivalence of distance between ranks 
makes valid the arithmetic manipulations 


involved in averaging. 


However, it is unlikely that many respond- 
ents will be presented with a list for ranking, 
where it happens that they perceive the rela- 
tive worth of all items as in any sense equally 
distant from each other. More commonly, res- 
pondents will feel strongly about one or two 
items as being much better than all the rest, 
and maybe about one or two items as being much 
worse than all the rest, but be indifferent 
towards all the others. 


When there is doubt about whether’ the 
equal-distance assumption applies in a given 
context, analysis should be limited to fre- 
quency counts only, with no adding or multiply- 
ing operations on the numerical ranks. 


Several other problems can arise in inter- 
pretation of responses, usually resulting from 
errors in administration of ranking questions. 
A respondent may consider two of the items 
equal for first choice, so he gives them both a 
score of one. Or, he may show a ranking for 
three items out of a list of six, and show 
nothing at. all »for other three... Or, ing 
complete misunderstanding of what he is asked 
to do, he puts a check-mark beside one (or more 
than one) of the alternatives. Sometimes these 
errors occur because the list of items is read 
out only once to the respondent. He is expected 
to remember all the items in order to be able 
to rank them. This excessive demand on the 
respondent's memory makes ranking questions 
generally quite unsuitable for any questioning 
procedure where the questions are just read 
out. The interviewer should be equipped with a 
prompt-card giving all the items that can be 
shown to the respondent as the question is read 


out to him. Clearly, in self-enumeration ques- 
tionnaires, the respondent can contemplate the 
list as long as he/she wishes. Conversely, in 
telephone surveys where nothing can be _ shown, 
ranking questions are in general quite unsuit- 
able. 


Choosing Between Open-ended and Closed-—ended 
Questions 


There is no simple set of rules that can be 
offered, in deciding whether a given question 
or set of questions would be better in open- 
ended form or in closed-ended. As has _ been 
described above, each type has its advantages 
and its limitations. The various factors having 
a bearing on the choice - the objectives of the 
question and of the survey as a whole, the 
positioning of the questions in the flow of the 
questionnaire, the likely extent of knowledge 
or experience of respondents, the extent of 
prior knowledge of probable answer-patterns 
from other sources, the availability and skills 
of respondents, interviewers, editors, coders, 
and data-capturers, the resources available to 
train them all, the method of data collection, 
and so on - must be considered. The relative 
importance of these factors varies from survey 
to survey, from topic to topic or even question 
to question within the survey. 


If most of the apparent advantages of one 
form of question appear to balance equally its 
disadvantages, then it is probably better to 
aim for the closed-ended version. Provided the 
categories are correct from the respondent's 
and the analyst's point of view, the ease of 
administering and processing closed-ended ques- 
tions seems often to be the deciding factor. 
Clearly, more developmental work is required to 
be able to draw up adequate closed-ended ques- 
tions, and questionnaires in the early stages 
of development often tend to have a higher pro- 
portion of open-ended questions than esta- 
blished regular surveys. Established surveys 
have the advantage of being able to learn from 
earlier waves or phases of development, in 
terms of creating (or improving) closed-ended 
questions. 


Attitude Scales 


This handbook does not propose to go deeply 
into problems of design of attitude scales. 
There is a voluminous literature on attitude 
measurement, with many examples of specialised 
scales used in psychological measurement (e.g., 
scales measuring sociability, powerlessness, 
and so on). A major caution to anyone planning 
to use attitude scaling questions is that the 
path to a successful scale is long. Repeated 
testing and refinement of a scale may be 
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required before the designer can be certain 
thate it Se ™valdd 1G Jevjo) that Pat “fig otreal iy 
measuring the attitude the designer thinks it 
is). Commonly, items in an attitude scale must 
be checked for adequate independence from one 
another (using such statistical techniques as 
factor analysis and discriminant analysis). 
Otherwise, the designer may unwittingly be 
measuring the same attitude element more than 
once by including, for example, phrase items in 
a scale that appear to be asking about dif- 
ferent things but are perceived by respondents 
as meaning effectively the same. Rather than 
trying to develop his own scale, a designer 
would be better to use one already developed, 
tested and validated by someone else. 


A second problem with all types of attitude 
questions is the extent to which such questions 
are subject to social desirability and polite- 
ness bias. Open-ended questions such as "What 
do you like about...?" followed by "And what do 
you dislike?" tend, overall, to receive many 
more comments about things liked than about 
things disliked. The reluctance of respondents 
to express themselves to an interviewer as 
disliking something can give a spurious impres- 
sion of "real" attitudes. Only when comparative 
analyses are made (e.g., ratio of liking/dis- 
liking comments for item A compared to item B), 
or respondents are forced into a preference 
choice, then asked reasons for preference, can 
a more accurate indication of real attitudes be 
obtained. 


In the case of attitude scale questions, the 
bias shows as a tendency for the average 
rating, overall, to be on the favourable side 
of any neutral mid-point. Again, any analysis 
must be comparative - e.g., the average rating 
this year compared to last year, or men rela- 
tive to women, and so on. An interpretation 
that an average rating is "favourable" indepen- 
dent of comparison with some benchmark or 
standard average rating, could be spurious and 
misleading. 


There are some widely used kinds of attitude 
scales. The next few pages describe the most 
common ones, for the most part known by the 
name of the person who first developed them. 


Thurstone Scale: 
Questions 


A Composition of Two-choice 


In a Thurstone scaling procedure, a subject 
is presented with a list of statements, each of 
which he is asked to either endorse or reject. 
Thus, a Thurstone scale is simply a collection 
of two-choice questions. An example follows, 
part of an attitude scale consisting of 
statements about war (adapted from Thurstone, 
1932, 


The details of how one translates responses 
into scores can be found in Oppenehim (1966). 
Areas of application of Thurstone scaling have 
included attitudes toward ethnic and cultural 
groups (MacCrone, 1937; Hinckley, 1932; Eysenck 
and Crown, 1949), social institutions (Remmers, 
1943), war (Dudycha, 1943; Ferguson, 1935), 


1. War is a futile struggle resulting in self-destruction 
2. We want no more war if it can be avoided without dishonour 


3. War is sometimes necessary because right is more important 


than peace 


4. War is glorious 


Likert Scale: A Composition of Multiple-choice 
Questions 


Like a Thurstone scale, a Likert scale is a 
collection of statements. The respondent con- 
siders each statement and reports how closely 
it reflects his own opinion. 


The chief difference between Thurstone and 
Likert procedures lies in the number of 
response alternatives for each statement. For 
Thurstone scaling, there are two; for Likert 
scaling there are usually at least five. The 
respondent indicates not only whether he agrees 
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Strongly 


agree 


It's not how you play the game, it's 
whether you win or lose 


An athlete is a gentleman who rep- 
resents his school 


Champions come in all sizes 


Winning isn't everything, it's the 
only thing 


be RE 


Guttman Scale: A Special Case of the Checklist 
Question 


A Guttman scale is a checklist question with 
special properties. Each of the items repre- 


political candidates (Beyle, 1932), and reli- 


gion (Granneberg, 1955). 


Instructions: For each of the following state- 
ments, please indicate with a checkmark whether 
you agree with it or disagree with it. 


Agree Disagree 


WU UU 
be ae 


or disagrees, but how much he’ agrees’- or 
disagrees. An example follows, a Likert scale 
based on statements about the glorification of 
sports (adapted from Jones' (1972) study of 
athletes). 


The output of a Likert scale is a numerical 
score, compiled from all the statements. The 
score .hasmits use: only)<n, agirelative esense, 
when compared to scores of other individuals in 
a population. The reader may consult Likert 
(1932). .for the - original, outline of. the 
procedures, with more recent summaries being 
found in Oppenheim (1971) Boyd, Westfall and 
Stasch (1981) and Scott (1968). 


Neutral Strongly 


disagree 


Agree Disagree 


UU L 
Bip el lea 
PE 
Bel Ei oe 


sents an increasingly strong expression of a 
Single attitude. The most well-known example of 
a Guttman scale was used in an attempt to 
measure peoples' attitudes toward racial 
groups, as shown below. 


"Into which of the following relationships 
would you be willing to accept a member of the 
ethnic group?" 

[__] Close kinship by marriage 

[__] Personal chums in the same club 

[___] Neighbours on the same street 

[__] Employment in same occupation 

ee Citizenship in the country 

(aaa Visitor to country 


None of the above. Exclude members of 
group from the country. 


fihe™ aspect “of interest is clearly «in the 
respondent's cutoff level. Because the items 
are ordered according to a decrease along a 
variable, we would expect a respondent to check 
all items below his/her cutoff level. 


The researcher may consult Guttman (1950) 
for the complete theory and scaling procedures 


House A House B 
Room sizes: generally large 


Number of rooms: 7 


Price: $60,000 Price: $70,000 
Location: city suburbs Location: 
Your-rank™''{. —] Your rank 


After respondents make several such choices 
in which the "levels" of several factors (room 
size, price, etc.) are varied, a statistical 
analysis is undertaken with the following 
outputs: 


- the relative importance of the different fac- 
tors (room size, price, etc.) in respondents' 
overall choices; 


- the relative value of individual levels of 
each factor (for example, how much large 
rooms are "worth" in comparison to small 
rooms) $ 


Room sizes: generally small 


Number of rooms: 


which characterize the technique, and will find 
selected applications in Podel and Perkins 
(1957), “Wallin (1953) and Clark and Kreidt 
(1948). 


Conjoint Measurement: A Composition of Ranking 
Questions 


Conjoint measurement is an attitude scaling 
technique whose potential is even yet being 
explored. The technique relies" -*on the 
respondent's ability to rank-order a set of 
alternatives, and from this rank ordering it 
attempts to assign quite precise measures of 
"value" to each alternative. For example: 


Suppose you were in the market for a house, and 
the three houses described below were avail- 
able. Based on these very brief descriptions, 
please indicate which house you would prefer 
most, second most and third most.Answer by 
putting a "1" beneath the description of your 
most preferred house, a "2" beneath the house 
preferred second most, and so on. 


House C 

Room sizes: generally large 
Number of rooms: 9 

Price: $85,000 


Location: outside city limits 


Your rank ha 


- an overall numerical value for each possible 
choice (Houses A, B and C), reflecting its 
favourability to the others. 


The evident advantage of the conjoint 
measurement technique is that it simulates 
lifelike situations in which consumers consider 
benefits in combination with each other - often 
having to give up some of one benefit in order 
to get more of another. 


Detailed analytic procedures for conjoint 
measurement are set out in Green and Rao (1971) 
and Johnson (1974). 


The Semantic Differential: A Composition of 


Rating Questions 
The semantic differential method of scaling 


attitudes involves presenting the respondent 
with a concept (e.g., the name of the attitu- 


Brand X Cigarettes 
Rating 


Cool 
Thick 
Mild 
Masculine 


Savory 


UUUUUU 
UUUUUU 
NUUUUU 
UUUUUU 
HUUU UU” 


Unusual 


The output of a semantic differential task 
is not one score for each respondent, but 
several. First, there is a score out of seven 
for him along each dimension. But statistical 
analysis (usually, factor analysis) of his 
several ratings results in a more compact set 
of scores, a set of scores which isolates the 
most important variables among all the ratings. 


dinal object) followed by rating scales along 
many dimensions. Each rating scale is 
"anchored" by a bipolar adjective, such as cool 
and hot, mild and strong, and so on. For 
example: 


Hot 

Thin 
Strong 
Feminine 
Tasteless 


Usual 


HULU UU” 
UUUU UU 


Semantic differentials are valuable when a 
respondent's attitude is expected to be too 
"complex" for measuring on a single dimension, 
and when comparisons among several objects are 
desired. By choosing bipolar adjectives 
judiciously, one can obtain insights into how 
specific feelings and perceptions contribute to 
the overall emotional appeal of a given object. 


Chapter 3 


QUESTIONNAIRE PRODUCTION 


Introduction 


Once a stage has been reached where it is 
felt that the proposed type and wording of 
questions are adequate, attention must’ be 
directed to the physical layout of the ques- 
tionnaire as a form, and printing of enough 
copies for all the interviews. 


Although these aspects are being discussed 
separately most designers will consider ele- 
ments of questionnaire production at the same 
time as they are reviewing wording. 


Layout 


There are several principles governing the 
physical layout of a questionnaire. Close 
attention to these at the design stage can 
serve to reduce error by interviewers, editors, 
coders, and key-operators or those manually 
transcribing answers from the questionnaires. 
The principles apply to the use of abbrevia- 
tions and punctuation, justification and posi- 
tioning of answer and instruction spaces, and 
typographical, colour and graphic elements. 


Abbreviations and Punctuation 


Abbreviations should be spelled out unless 
they are very common ones all respondents are 
likely to understand, or ones that have already 
been well defined earlier in the interview. 
Critical words may be underscored or printed in 
bold letters to ensure uniform emphasis’ by 
interviewers, and uniform interpretation by 
respondents. 


"Now we'd like to talk about last week. Did you 


ocr: 


Writing last week in bold face helps emphasize 
a change in time reference. 


Commas, colons and dashes, improperly used 
may cause a break in the interviewer's verbal 
flow and lead the respondent to formulate an 
answer prematurely, before the question is 
Finished. 


Response Categories 


Answer spaces for questions requiring short 
answers (e.g., check-mark or numerical) can 
usefully be margin justified rather than being 
strung out across the page. It is easier to 


edit, code and tabulate answers when scanning 
is kept to a minimum by aligning spaces. As a 
general rule when form design has an inner 
logic which is readily grasped, data capture 
operations will be accomplished more rapidly 
and with fewer errors. 


44. (a) In this job were you mainly: 


02 fee} working for wages, salary, tips or 
commission? ; 
Go to Question 45 


03 C working without pay for a relative 
in a family farm or business? 


04 i= self-employed without paid help? Contindenwith 
05 (_] self-employed with paid help? Question 44(b) 


® (b) If self-employed, was your farm or business incorporated? 


06 CJ] No 


07 C Yes 


45. (a) In how many weeks did you work during 1980 (not including 
S housework or other work around your home)? 


Include those weeks in which you: 


© worked full-time or part-time; 
| ® were on vacation or sick leave with pay; 
e were self-employed. 


os [_] None > Go to Question 46 
OR 


09 | | Weeks 


(b) During most of those weeks, did you work full-time or part-time? 
Mark one box only 

10 L_] Full-time 

11 = Part-time 


Sometimes it may not be practical to allow 
the answer categories to be justified to the 
margin of the page. If there is a large number 
of list-type questions, putting them all one 
under the other would lengthen the question- 
naire yet leave much of the space completely 
unused. 


In these circumstances, it may be better to 
spread answer-categories across the page, in 
the form of a matrix or a table or allow a 
column (say, 21 cm or 31 cm wide) at the right 
margin, headed "For Office Use Only", with 
coding boxes into which answers can be coded, 
by interviewers or by coding clerks. 


Even if capture takes place directly from 
the matrix, boxes are generally needed for 
entry of codes for open-ended narrative ques- 
tions (e.g., occupations). An example follows 
from the 1981 Census of Canada. Although this 


particular survey was self-completed by the 
respondent, the layout could equally well have 
been used in an interviewer-administered 
questionnaire. 


41. NOTE: Questions 41 to 44 refer to your job or business last week. 
If none, answer for your job of longest duration since January 1, 
1980. If you held more than one job last week, answer for the job 
at which you worked the most hours. 


(a) For whom did you work? 


Name of firm, government agency, etc. 


ie Cae IY. tir EY 5 ] 


Department, branch, division, section or plant 


(b) What kind of business, industry or service was this? 


cd eee 


Give full description. For example, paper box manufacturing, 
road construction, retail shoe store, secondary school, dairy farm. 


42. Atwhat address did you work? If no usual place of work, see Guide. 
Mark one box only 

© [_] Worked at home (includes living and working on the same farm) 

[_] Worked outside Canada 

[_] Worked at address below (please specify) - 


5. spleen ] 


ae 
< 


If street address is not known, give the building name, shop- 
ping centre or street intersection, etc. 


aaa i 


Important: /f you worked ina suburban municipality within a 
large urban area, specify that municipality, not the main city. 


nel 


ie. a 


Province or territory 


Space 


Ideally the form should not be too crowded 
or cluttered. If it is, interviewers/respon- 
dents will miss instructions or questions, or 
record answers confusingly (e.g., spread over 
two answer spaces). Roughly, one centimentre of 
space should be allowed for two written digits 
in a numerical answer. Enough room should be 
allowed for answers to questions requiring a 
narrative. If there is too little space, inter- 
viewers/respondents will be tempted to edit or 
abbreviate the answers or to be selective in 
what they record eliminating parts which may be 


important to users of survey results. In self- 
completion questionnaires respondents may 
abbreviate their answers to fit the space 


available, leading to difficulties of interpre- 
tation in coding the response. 
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On the other hand, if a questionnaire is 
used over a long period of time for a continu- 
ing survey, interviewers and other users of the 
form will become used to a layout that may at 


first sight appear cluttered. The following 
example from the Current Population Profile 
(1982) illustrates the latter point. This 


questionnaire is interviewer-administered, both 
in person and by telephone. 


Answer Spaces 


Answer spaces should be clearly identifiable 


with the question to which they apply. The 
following example illustrates the kind of 
confusion that arises when the answer spaces 


are crowded together: 


under 20 20-24 35-49 50-64 
65 or over. 
An interviewer could easily enter ~ the 


check-mark to the right rather than to the left 
of the correct category. A better layout would 
be to list each class and corresponding answer 
space on a separate line. 


[| under 20 
[-aanepeos24 
ieee 25249 
[_] 50-64 
{| 65 or over 


If this is not feasible, 
would be an improvement: 


[_]under 20 [___] 50-64 
[_] 20-24 [| 65 or over 
[suas} 25-49 


If the answer spaces are listed one under 
the other, and margin justified, errors may 
still occur if the space between the question 
and the answer space is too great. The inter- 
viewer's or respondent's eye could slip up or 
down a line in the middle of the following 
layout: 


the following layout 


Under 20 
20-34 
35-44 


45-54 


UUUU 
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| & Treasury Board of Canada Conseil du Trésor du Canada 
Secretariat Secrétariat 


SURVEY OF PUBLIC SERVICE EMPLOYEES 


(confidential when completed) 


This questionnaire is to be filled out only by Indigenous people, disabled persons and members of visible minorities. Please 
read questions 1 to 3 carefully to determine whether you belong to any of these groups. 


This information is supplied on a voluntary basis, and will not be used in relation to any personnel decisions which relate 
to individuals. It will be augmented with data from other files in order to analyse and monitor the situation and progress of 
the survey groups in terms of such characteristics as regional and occupational distribution, training and mobility. This informa- 
tion will be retained for a period of 10 years under Personal Information Bank TB-C-P110. Under the Privacy Act, you have 
the right of access to, and protection of, information relating to you. 


1. Please check the appropriate box, if you are: 


1 L] Inuit 2 L] Metis 3 4 Non-Status Indian 4 Ls Status Indian 


. Do you have a permanent disability which relates to: 
(check more than one, if appropriate) 


Co-ordination 
"1 a or dexterity 


12 LJ Mobility uk! ie Speech 14 # Epilepsy 15 oe Mental retardation 


Vision B 161 _|Blind 17 |_| Partially sighted Hearing B 181] Hard of hearing 19 |_| Deaf 


. Do you consider yourself to be of: (check more than one, if appropriate) 
i) African origin (Black) 41 LJ Includes Caribbean and Canadian/American 
ii) Asian origin 


‘nme | 


42 Bangladeshi 43 es Cambodian 44 |_| Caribbean 45 LJ Chinese 46 [ Indian (India) 


47 Japanese 48 aa Korean 49 |_) Laotian 50 LJ Pakistani 5) LC] Filipino 


52 Vietnamese 53 LJ Other (please specify) 


iii) Oceanic origin 54 | Jincludes Polynesian, Melanesian, Micronesian, etc. 


iv) Any combination of African, Asian or Oceanic origins 55 LJ 


If one or more of the above applies to you, please complete the section below, and mail this questionnaire to the 
Treasury Board Secretariat, Ottawa, using the envelope provided. 


4. Family name 6. Social Insurance Number 
(optional) 


. Department (e.g., Department of National Defence, Transport Canada, etc.) 8. Classification 
Group Sub-group Level 


. Place of work: (please check one only) 


35Llont. 12LI Ms. 13 LIN. 10 [ ]Quebec 481] Alta. 


ia 


47 Sask. dil aluealy Pe Est 60: Yukon 59 1G. ' Outside Canada 


OFFICE USE ONLY 


TBC 330-249 (Rev. 85/2) 


TB/CT-REG. B8061 (FRANCAIS AU VERSO) 
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BW statistics canada Statistique Canada CURRENT POPULATION PROFILE CONFIDENTIAL when completed 
Docket No 2 ee ee ee Survey date 3 Assignment No 4 06 
HRD page - line No Given name Mo Yr Surname 1 FORM NO 


5/ |] 6L. 7| 


10 INTERVIEWER CHECK ITEM 


20 HAS... LIVED IN ANY OTHER PROVINCE, TERRITORY, 
If age 00 to 14 in item 33 on A OR OTHER COUNTRY SINCE JUNE 1, 1976? 
cs) Go to 48 


‘ Vesue) No 2) Go to 48 
Otherwise cn tee oe As ls C) Go to 17 


LANGUAGE ; MIGRATION HISTORY 


11. WHAT IS THE LANGUAGE ... FIRST LEARNED IN CHILD- | 21. IN WHICH PROVINCE, TERRITORY, OR OTHER COUNTRY 


HOOD AND STILL UNDERSTANDS? (Mark al/ that apply) DID... LIVE ON JUNE 1, 1976? 
Bnglishweetene od akectiet tee 10) eee 
Enter code 
2 
Rrench) connate acento aes © i: 
- 22 TO WHICH PROVINCE, TERRITORY, OR OTHER COUNTRY 
Other ae a en. Cee O DID ... FIRST MOVE AFTER JUNE 1, 1976? 
| 
12 CAN...SPEAK ENGLISH OR FRENCH WELL ENOUGH TO 
CONDUCT A CONVERSATION? Bal EnteR Code 
NO — Neither English nor French ...... a) 
2 23 WHENDID...MAKE THIS MOVE? 
YES —EnglishionlWa ey cise cual pees O Go to 14 


3 
Wass lamomManadoarmaduane As @) Mo. Leas Meas Yate 


YES — Both English and French ....... at) Go to 13 
[ 24 TO WHICH PROVINCE, TERRITORY, OR OTHER COUNTRY 
13. IN GENERAL, WHICH OF THESE TWO LANGUAGES DOES DID... MOVE NEXT? 
_.. PREFER TO SPEAK? ea Enter node 
1 
Enaligne sot: tata rane e ce) /f code 99 (No other moves) go to 34 
x6) 
Frenechins) tie Mnns pam 25 WHEN DID...MAKE THIS MOVE? 


NGitheNn <a peamcichaie Ge Recetas 1) Werle 
1@) Mo. Vite 


fidibeetorence meme 4 ‘© 26 TO WHICH PROVINCE, TERRITORY, OR OTHER COUNTRY 
DID... MOVE NEXT? 
14. WHAT LANGUAGE DOES. ..SPEAK MOST OFTEN AT HOME? SS 
(Mark all that apply) Ez If code 99 (No other moves) go to 34 


Englishes caw: sete ct eectitetecss 1 |= 
Kench wee ee eet © 


3 
ae ae O ei Mie 


28 TO WHICH PROVINCE, TERRITORY, OR OTHER COUNTRY 
DID... MOVE NEXT? 


27 WHEN DID...MAKE THIS MOVE? 


15 HAS... (EVER) ATTENDED A UNIVERSITY, COMMUNITY 
COLLEGE OR OTHER POST-SECONDARY INSTITUTION AS Bia Enter code 
A FULL-TIME STUDENT? If code 99 (No other moves) go to 34 
i! 2 
Yeon) No ©) Go to 20 29. WHENDID...MAKE THIS MOVE? 
16 WHAT IS THE HIGHEST LEVEL OF EDUCATION... Mo eaters Yr 
COMPLETED? 
ean Eade 30 TORN eo ROMI ee TERRITORY, OR OTHER COUNTRY 
If code 99 go to 20 we ; 
as) Enter code 
17. IN WHAT YEAR WAS . . . ‘S LAST DEGREE, DIPLOMA OR If code 99 (No other moves) go to 34 


CERTIFICATE GRANTED? 


31 WHEN DID...MAKE THIS MOVE? 


Let LN ven re Lui 


i! 
18 IN WHICH PROVINCE, TERRITORY, OR OTHER COUNTRY 32 TO WHICH PROVINCE. TERRITORY. OR OTHER COUNTRY 
WAS THIS DEGREE, DIPLOMA OR_- CERTIFICATE DID... MOVE NEXT?_ ‘ 


GRANTED? 
a) Enter code 
RE Pas ee 3 /f code 99 (No other moves) go to 34 


33. WHEN DID...MAKE THIS MOVE? 


aa ee 
Enter code Use an ANNEX if more than 6 moves 
Otherwise go to 34. 


19 WHATWAS...’S MAJOR FIELD OF STUDY? 


8—5400-—61.1: 11-6-82 TB/CT—REG. B102550 
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If it is not possible to position the cate- Interviewer Working Space 
gory names close to the answer spaces, a dotted 
line (leaders) connecting the answer and the If numeric units are to be converted to 
recording space could help cut down line other units by interviewers, working (or 
slips. ; "Office Only") space should be allowed for this 


on the questionnaire. This also applies if 
interviewers are to total a set of numerical 
answers as a first editing check, before 
Finishing the interview. For example, in expen- 
diture surveys, income and expenditure for the 


PIOAHD FOVER datas otters 6 Baw ce ee ss 2 ule ate ere fo 2G reporting period must be the same (or within 


55-64 eoeeoereeereoee eee eee eereeoeoeeeeree ee eevee 


acceptable margins of difference). If they are 
not, the interviewer must often re-question the 
respondent to determine whether income or 
expenditure items have been overlooked or 
recalled inaccurately. Expenditure survey ques- 


DOT cs CIAO Witkerenetoncs a6 Cotton eee lelotote © etene Stele ove leis: ole 


Refused eoeeveceeeeeroe eee eee eee eeeeeeaeeeeve 


For open-ended narrative questions, space tionnaires must therefore provide working space 
into which codes can be entered (or better, for interviewers to enter totals of expendi- 
boxes) should be provided at the right margin tures and incomes, and calculate differences 
at the end of the. last line. between them. 


| te Statistics Canada Statistique Canada 


CONFIDENTIAL 
when completed 


TOURISM ATTITUDE AND MOTIVATION STUDY 
| 
14. | WOULD LIKE TO KNOW HOW IMPORTANT EACH OF THESE FACTORS WERE IN YOUR DECISION TO TAKE YOUR MOST 
RECENT VACATION OR PLEASURE TRIP. PLEASE READ OVER THIS LIST OF STATEMENTS AND MARK HOW IMPOR- 
TANT EACH WAS TO YOU. (GO TO PAGE 6, HAND QUESTIONNAIRE TO RESPONDENT TO COMPLETE ITEMS 34 AND 35.) 


ees aa Ba ree Ste aie : 


July, April, January, October, 
August, May, February, November, 
September 1983 June 1983 March 1983 December 1982 


Yes No Yes No 
15. WERE ANY OF THESE TRIPS TAKEN 
DURING THE MONTHS OF......... 6 eG) Ses) AY ad @ 


No. of Nights No. of Nights No. of Nights No. of Nights 


FOR EACH ‘YES’ ASK Q. 16, 17 AND 18. 


16. HOW MANY NIGHTS WERE YOU AWAY 
ON THE TRIP OF THE LONGEST DURA- 
TION IN... (MONTHS)? 


Enter code Enter code Enter code Enter code 
for month for month for month for month 
17.1IN WHICH MONTH DID YOU START Geren Boma os ye pes! 
THIS TRIP? 
City or town: City or town: City or town: City or town: 
| Prov, or state ae or state Prov. or state a) Prov. or state 
18. WHAT WAS YOUR DESTINATION ON | 
THIS TRIP? (FURTHEST POINT AWAY Country: Country: Country: Country: 


FROM HOME.) 


For office use only | For office use only | For office use only | For office use oniy 


Rec ell Mp p al ML sl rete 


Differentiation 


The questionnaire can be designed so that 
one kind of answer space, e.g., a small circle 
is used whenever a check-mark is called for, 
as in "yes/no" questions or selections from 
lists, and a different kind, e.g., square boxes 
whenever numbers are called for, as in ages, 
numbers of times something has been done in a 
time period. This can be useful to editors 
checking each questionnaire before data cap- 


ture. They can learn to expect checks in cir- 
circles and numbers in boxes. In the preceding 
example, from the Tourism Attitude and 


Motivation Study (1983) circles were used for 
check-marks and boxes were used for numerical 
answers. 


The use of heavy solid lines (known as 
"rules") as a visual aid to the interviewer can 
be helpful between questions or sets of ques- 
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tions. The same applies to column-divisions, if 
they are to be used. 


An illustration of lines clearly separating 
questions can be seen in the earlier examples 
from the 1981 Census of Canada. 


Shading or cross-hatching of those parts of 
the questionnaire that are not to be used for 
answers that are to be captured, can be a use- 
ful device. In this way, the right margins or 
other parts of the questionnaire, from which 
responses are to be keyed will stand out. An 
example is given below from a wildlife survey 
done by Statistics Canada for the Canadian 
Wildlife Service, Environment Canada. All ques- 
tions and instructions to respondents in this 
self-enumeration survey were on a_ coloured 
shaded background. Only the squares and circles 
for responses to be captured used a white back- 
ground. 


D. HUNTING WILDLIFE 


35. HAVE YOU EVER HUNTED WILDLIFE? 


Yes 23) 


y 


36. DURING 1981, DID YOU HUNT WILDLIFE? 
Yes ry) 


v 


37. IN TOTAL, ON HOW MANY DIFFERENT DAYS DID YOU HUNT WILDLIFE IN 1981? 


i Nee N Wier 
38. DURING 1981 DID YOU HUN 


Yes a @)) 


T WATERFOWL? 


No ?C)__»Go to Question 48. 


39. ENTER THE NUMBER OF DAYS BESIDE THE LOCATION(S) 


WHERE YOU HUNTED WATERFOWL IN 1981. 


Days 
Newfoundland 01 ez Saskatchewan 08 
BiEsle 02 fee Alberta 09 
Nova Scotia 03 ra British Columbia 10 
New Brunswick 04 ral Yukon 11 
Quebec 05 hel N.W.T. 12 
Ontario 06 [se] 
Manitoba 07 Be Outside Canada 13 


EF] EI ICIs 


48. DURING 1981 DID YOU HUNT 


2 
No O — Go to Question 78. 


2 
No O —- Go to Question 78. 


Days 


—————+ Go to Question 38. 


: On, f 
—— abe) a Sa A 


OTHER BIRDS? 


No ae) ——¥ Go to question 58. 


Yes :@ 


Y 


49. ENTER THE NUMBER OF DAYS BESIDE THE LOCATION(S) 


WHERE YOU HUNTED OTHER BIRDS IN 1981. 


Days Days 
Newfoundland 01 re Saskatchewan 08 ee 
PeeAle 02 Ez Alberta 09 al 
Nova Scotia 03 te British Columbia 10 al 
New Brunswick 04 fein Yukon 1 wil 
Quebec 05 ee N.W.T. 12 Ld 
Ontario 06 ial 
Manitoba 07 ee 13 al 


Outside Canada 


Note also the use in the preceding example 
of visual illustrations. These or other 
graphics can be used on cards shown to respon- 
dents as aids to answering questions - ffor 
example, where the respondent is expected to 
make a choice from a list that is long. They 
also help maintain respondent interest in the 
survey. 


Colour 


If coloured paper is available, it can be 
used to separate sections which apply to sub- 
sets of the population. For example, in a 
question on employment status, answer classes 
as a general screening question could include 
"now working", "not working and looking for a 
job", "housewife", "student", and so on. The 
respondents answering "now working" could be 
directed to (GO TO BLUE PAGES) where the blue 
pages include only questions relevant to some- 
one currently working. Similarly respondents 
"not working and looking for a job" could have 
the instruction to (GO TO PINK PAGES), where 
the pink pages would contain questions only for 
someone unemployed. 


Light pastel shades should be used, since 
black print on dark-coloured paper may be hard 
for interviewers to read. Similarly, print in 
colours other than black, e.g., light blue or 
light green on light paper may be hard to read, 
and may not reproduce well with certain kinds 
of duplicating processes. 


Another option is to use colour over-print- 
ing at the corners of the first page of each 
section. While the rest of the questionnaire 
can be printed in black on white, these 
coloured corners (using a different colour for 
each section) provide an easy visual identifi- 
cation of each section. 
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Numbering of Questions 

Questions should be numbered sequentially 
throughout. And in questionnaires which are 
divided into sections, a multiple numbering 
system will be useful. It will consist -of .a 
primary numbering system for sections and a 
secondary system for questions within sec- 
tions. Thus each question will have a distinct 
number, somewhat like an apartment in an apart- 
ment. sbuildrng,. tee; Apts 45:de-thesfifth apt. 
on the 4th floor similarly Question 45 is the 
fifth question of section 4 and not the 45th 
question of the questionnaire. 


Instructions 


Instructions to interviewers or respondents 
should be placed directly above the question(s) 
concerned. In this position, they are most 
likely to be read and acted upon. The next best 
place is at the beginning of a section of the 
questionnaire. Third best is at the beginning 
of the questionnaire. Here, however, they may 
be read once only, then forgotten. The worst 
place is in a separate interviewer's manual. If 
it is ever read, it is unlikely to be referred 
to again, especially not in the middle of an 
interview. The end of a group of questions 
should be signalled clearly, usually by another 
instruction (e.g., "ASK EVERYONE"). 


Many business surveys require’ detailed 
instructions. Because of space requirements 
there often is no alternative but to give these 
instructions on a separate sheet or in a sepa- 
rate booklet. Where this is the case, instruc- 
tions should appear on the questionnaire 
directing the interviewer or respondent to con- 
sult the appropriate section of the instruction 
booklet. 


REPORTING INSTRUCTIONS AND AUTHORITY: Completed questionnaires must be returned within 60 days of receipt. The enclosed Guide is designed to assist in 
the completion of this report. Instructions are numbered to correspond to the numbers on the lorm. Please keep a completed copy of this Form for future reference. 


Collected under Authority of the Statistics Act. 


INFORMATION SHARING AGREEMENTS 


To reduce response burden and to ensure more uniform statistics, Statistics Canada has entered into agreements with various government departments and agencies for 


the sharing of data: 


Under Section 10 of the Canada Statistics Act, with the provincial statistical agencies of Newfoundland, Nova Scotia, Quebee, Ontario, Manitoba, Saskatchewan, 
Alberta and British Columbia in respect of establishments located within the boundaries of their respective Province. The Statistics Acts of these Provinces include 
substantially the same provisions for confidentiality and penalties for disclosure of confidential information as the Canada Statistics Act. 


Under Section 11 of the Canada Statistics Act, with the Prince Fdward Island Department of Industry and Commerce, Prince !'dward Island Department of Develop- 
ment, and the New Brunswick Economics and Statistics Branch of the Department of Finance, for all establishments located within their respective Provincial boundarics. 
Seetion 11 agreements shall not apply to your 1984 Census of Manufactures report(s) if an authorized officer or person of your company objects in writing to the 
Chief Statistician and mails that letter to the Manufacturing and Primary Industries Division of Statistics Canada together with the completed questionnaire. Please 
specify those agencies or departments listed above from which data shall be withheld. 


1.9 REPORTING YEAR ~— Report must cover your most recent financial year ending between April 1, 1984 and March 31,1985. 


i 


hells ps8 bes 


from er L | 


Day Month Year Day Month 


Instructions should be in a clearly diffe- 
rent type face or type size, or in bold type or 
capital letters, or in a box, so -that inter- 
viewers and respondents will not confuse them 
with the questions. They should be short and 
clear. 


A dwelling is a separate set of living quarters with a private 
entrance from the outside or from a common hallway or stair- 
way inside the building. This entrance should not be through 
someone else’s living quarters. 


7. Enter the name of the person (or one of the persons) who lives here 
g and is responsible for paying the rent, or mortgage, or taxes, or elec- 
pee tricity, etc., for this dwelling. 


, 07 
01 if a] 
(ast name! nn Sn Given name and initial  —__ 
This person should answer the following questions about this dwelling. 
NOTE: If no one living here makes any such payments, mark 
here [ and answer the dwelling questions yourself. 
Arrows and boxes can often help as an 


adjunct to skip instructions. It can sometimes 
be useful to repeat questions that are to be 
asked of everybody, on both sides of the skip, 
just to make the questionnaire easier for 
interviewers to follow. 


Instructions ought to be consistent with the 
questions to which they apply. In the inter- 
viewer's control docket for the Census of 
France in 1946 confusion was introduced by the 
lack of clarity and consistency between the 
questionnaires and the interviewer's instruc- 
tions. The section on housing began with the 
question: 


Dwelling: 


Occupied? (yes or no) 
Vacant? (yes or no) 


A presentation which suggested that there 
might be a third status, i.e., temporarily 
unoccupied). Further on in the docket there 
appeared an instruction: (use one line for each 
dwelling that is occupied, unoccupied or 
vacant) which could be interpreted as reinforc- 
ing the three-class classification. However, 
the interviewer's instructions spoke only of 
dwellings that were "vacant, that is unoccu- 
pied", suggesting clearly that only two classes 
were intended. 
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Language 


For national surveys where questionnaires 
are to be printed in two languages, and on both 
sides of the paper, the most convenient layout 
is to print separately back-to-back in a "tum- 
ble" format. This is practiced only if printing 
on large (e.g., 30 cm by 42 cm) paper folded 
once (to make 21 cm by 30 cm pages) and 
saddle-stitched (i.e., stitched or stapled in 
the fold or "gutter"). Quite commonly, 
questionnaires are printed on 21 cm by 30 cm or 
21 cm by 35 cm stapled only in an upper corner, 
or stitched (stapled) two or three times down 
the side. If the questionnaire is to be printed 
Single-sided and in both languages, with one 


language followed on the next line by the 
other, then the second language should be in a 
distinctly different type face (e.g., in 


italics), or in bold type. Alternatively, the 
questions can be laid out in a columnar fash- 
ion, but providing a common central answer 
space for both languages. 


If more than two languages are involved, for 
example in surveys among ethnic minorities, it 
is probably better to use separate self-con- 
tained questionnaires for each group. 


Consistency 


Once a layout pattern has been arrived at, 
it should be used consistently throughout the 
questionnaire. Changing the layout may only 
serve to confuse the interviewers, editors and 
coders. For example, if instructions are in 
capital letters, or boxed, or however they are 
differentiated from the questions themselves, 
they should be shown the same way on all ques- 
tionnaires and control sheets. 


Data Coding and Capture 


The present study assumes that data for all 
except the smallest surveys will be processed 
by computer. If adequate computer facilities 
are not available, the benefits in terms of 
speed, cost, accuracy and flexibility 
are so overwhelming that every effort should be 
made to rent them on a temporary basis or 
contract to have the processing carried out by 
a private computer service. Only the most 
basic counts should be carried out by hand. 


If it is impossible to obtain computerised 
processing and there is a requirement to pro- 
vide quick summary statistics, the question- 
naire should be designed so that summary totals 
can be produced directly from it. Thus wherever 
there is a listing of items, a summary total 
space should also be included and interviewers 
should be trained to make these calculations as 
soon as the interview is completed. If values 


must be calculated, formulae for calculating 
values ought to be provided directly on the 
form. 


Consideration must be given to the capture 
of the data for subsequent processing at the 
design stage or the designer will certainly be 
faced with a long expensive error-prone task of 
manually coding and possibly transcribing data 
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from the questionnaires, pre-coding, that is, 
showing the codes on the printed questionnaires 
is an obvious convenience for keyers, de- 
creasing the possibility of keying an answer 
into a wrong field. If answers are to be posted 
onto 80-column sheets showing the field codes 
on the questionnaire can indicate to clerks 
where they are to write the answers on the 
sheets. The code numbers should be printed 


onto large sheets for the remaining processing 
operations. 


relatively inconspicuously, in order not to 
confuse interviewers, or respondents if the 
questionnaire is to be completed by them. 

It is essential to consult early, regularly, 
and often with the processing staff, to design 
the questionnaire for rapid data capture. One 
way of ensuring that the concerns of data 
capture are addressed is to make the individual 
responsible for this aspect of the survey a 
permanent member of the questionnaire design 
team. In’ fixed field coding, each unit of data 
(e.g., each question for which an answer must 
be captured) is assigned a number of columns on 
a record or "positions" sufficiently great to 
accept the longest possible answer. The record 
length for each questionnaire will be fixed 
equal to the total number of columns assigned. 
Once a column is assigned it must contain a 
value. If there is a blank answer, then the 


Coding Systems 


There are two basic systems of coding - 
fixed field coding and source coding. 


Printing Codes on the Questionnaires 


If data are to be processed by computer, 
codes for the fields into which answers are to 
be keyed should be printed directly on the 
questionnaire. If keying is to be done directly 
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HRD page - line No. Given name Mo Yr Surname 


20. was THAT COURSE... (Mark only one) 


CONFIDENTIAL when completed 


1 FORM NO. 06 


THIS SURVEY CONCERNS ANY COURSES, CLASSES OR INSTRUCTION () 
.. . HAS TAKEN DURING THE PAST YEAR. THESE WOULD INCLUDE AD LACad ei ClCOULSO.7 ier gtk ae = teats tA en. eek 

INSTRUCTION TO IMPROVE JOB SKILLS, UPGRADE ACADEMIC 1G) 
QUALIFICATIONS, FOR PERSONAL DEVELOPMENT OR FOR AVjOU-TELRLCURCOUISC? —) Eieis atthe eae e- ie en een 

RECREATION AND LEISURE. Kay 
A hobby: craft, of recreation Course? 2s 5 + = «ae 

10. WAS ... A FULL-TIME STUDENT AT A SCHOOL, COLLEGE OR hd) 
UNIVERSITY DURING 19837 A personal development, general interest course? ....... 


1 2 5 
Yes O No ©) Go to 11 Otfier ap i16.. 93. Hees. ole belli): ee oe @ 
READ iD 
Dontnions 20). .c0es FOSS .. Jeo... Dee. 47... 


IN THE FOLLOWING QUESTIONS DO NOT INCLUDE 21 WHO ORGANIZED THE LAST COURSE TOOK, THAT IS 
WHO OFFERED THE COURSE? 


ee ler| Code 


11. purine 1983, DID... ENROLL IN ANY COURSES TO 22. WHICH WAS THE MORE IMPORTANT REASON FOR TAKING THIS 


CLASSES OR COURSES THAT WERE PART OF... 
FULL-TIME COURSE OF STUDIES. 


UPGRADE ACADEMIC QUALIFICATIONS? COURSE, TO 
©) Ge) we | 
Yes No Don’t know Imptove: jobvepportunities?, 23.554... vewee@e + sealer chee @ 
1 2. DURING 1983, DID... RECEIVE ANY INSTRUCTION OR ‘ce, 
TRAINING TO UPGRADE JOB SKILLS, INCLUDING COURSES Or for personal interest and development?............. 
TAKEN AT WORK? 3 on 


ie) ‘ey © DOMOKNOWT > Sees A aE i SE Seen Cash. oe 
Yes No Don’t know 23 


HOW MANY HOURS OF INSTRUCTION WERE THERE EACH WEEK? 
1 3. DID... ENROLL IN ANY CLASS OR COURSE FOR PERSONAL 


88 
INTEREST OR TO DEVELOP PRACTICAL KNOWLEDGE SUCH AS AN wade Hours Don’t know 
ART OR CRAFT CLASS, DURING 19837 24. HOW MANY WEEKS DID THE COURSE LAST? 


1 2 3 Ap 
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columns must be "packed" with zeros or some 
such no-value symbol. Even partly completed 
questionnaires will need a full record and 
questionnaires will need to be coded with "O"'s 
in each position which has been left blank by 
the interviewer or respondent. What this system 
saves in design time, it spends in processing 
as each record must be fully read at the time 
of tabulation. 


each unit of data is 
assigned a source code - a sequential number 
which has no regard to the length of the 
largest potential answer. Often this number is 
the number of the question itself. In source 
coding, the keyer ignores blank fields. 
Resources are economized in data capture. For 
example, in a survey with, say 1,200 fields, an 
average of only 200 or so may need to be cap- 
tured for each respondent. Resources are econo- 
mized even more significantly in data process- 
ing where the computer usually deals with 
shorter records, that is, records not "packed" 
with no-value symbols. One limitation of source 
coding is that many computer programmes for 
statistical use, e.g.,~SPSS were: developed at~a 
time» of. fixed field coding. There must. bea 
computerised step of conversion from source 
coding to fixed field coding to use these 
programs. The decision as to which system to 
adopt is a specialised one, but it must be made 
prior to printing the questionnaires. 


In source coding, 


Location of Code Lists 


If interviewers are expected during the 
interview to code from a separate list, i.e., 
one that is not part of the question itself 
(example: region or occupation), the codes 
should be provided right on the same page of 
the questionnaire, for example, at the bottom 
of the page. 


Maintaining the code list separately, such 
as in the interviewer's manual, is not conve- 
nient and is likely to lead to error. If the 
list is too long to put right on the question- 
naire, the next most convenient location is on 
a separate code card that may contain codes for 
a number of questions. 


Coding "Other (SPECIFY)" 


If the questionnaire contains answer lists 
that end with "other (SPECIFY)" a decision must 
be made on how to capture answers to the 
"SPECIFY": are all to be captured together or 
separately? If they are to be captured indivi- 
dually, a code must be allowed for each. A 
checklist of leisure activities might yield as 
many as 20 further activities beyond those 
already listed and it might be necessary to 
allow for a two-column field or for several 
extra fields to capture all answers. It is more 
probable however that limits on record lengths 
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will dictate some decision, however arbitrary, 
on, procedures to limit the number of extra 
items to be captured. 


Coding “No Answer" and “Not Available" 


"No answer" and "not available" classes can 
often be assigned the same code. Once assigned 
the code for this class should be the same for 
all questions. In this way, confusion among 
coders, editors, keyers and analysts is de- 
creased. In some circumstances, a "don't know" 
answer can be given the same code, although it 
is generally :desirable ,to,.give*it... another 
code, as conceptually, a "don't know" answer is 
different from a "no answer" and "not avail- 
able". 


Similarly, "yes" and "no" should be treated 
consistently throughout the questionnaire (for 
example, "1" and "2", although some researchers 
prefer "1" and "0", considering "0" more symbo- 
lically representative of "no" than "2"). 


Printing 


Questionnaires should be attractive looking, 
especially if respondents are to complete them 
independently. Care must be taken to use good 
quality presses and paper. If the questionnaire 
is to be typed and then photocopied or repro- 
duced»sin an offset -process, or through. a 
Gestetner or similar process, a good electric 
typewriter is desirable, to give an evenness of 
quality to letters, boxes, circles, pairs of 


brackets, ete. Since few electric. typewriters 
can reproduce such symbols as arrows or 
circles, these will likely have to be drawn by 


hand. It may be feasible to acquire sets of 
symbols that can be cut and pasted onto pages 
being prepared for the camera in the offset 
process, or may be possible to access computer 
facilities that include a text-editing program 
with symbols, and these can be used to prepare 
camera-ready copy. 


In certain situations, 
supply may be a factor 
length of the questionnaire, and whether a 
"schedule" format should be used. If there is 
any doubt, the questionnaire designer should 
determine early in the planning process just 
what printing or duplication processes will be 
available. Shorter questionnaires dictated by 
limited printing facilities may require better 
trained interviewers and clearer instructions, 
i.e., better design. 


limited printing or 
in determining the 


Paper Quality and Size 


Questionnaires should be printed on sturdy 
paper, since they will generally be handled a 
number of times by printers, shippers, supervi- 
sors, interviewers, respondents, editors, 


coders, key-operators, file clerks or storage- 
clerks. The physical size and bulk of the ques- 
tionnaire should be convenient for interviewers 
and/or respondents to handle in different 
interviewing circumstances. 


Single- vs. Double-sided 


Questionnaires are often printed only on one 
side of the paper commonly because of the 
printing equipment available. If necessary, the 
reverse side may be used for notes or for over- 
flow answers to questions although this is not 
a recommended practice. Single-sided printing 
may also be more convenient in the interview 
Situation. A disadvantage is that question- 
naires take twice as many sheets of paper com- 
pared with double-sided printing, increasing 
the bulk that must be handled and shipped. 


Quantities 


Questionnaires should be printed to provide 
for the maximum number of interviews and to 
have as extra for: 


- training - trainers and trainee interviewers 
need copies to work on, such as in dummy or 
mock interviews, or in test interviews with 
"real" respondents; 


~- spoilage or loss, or when additional respon- 
dents may be found. In some interview situa- 
tions, it may not be possible to tell 
accurately in advance how many respondents 
each interviewer will find (e.g., surveys of 
all members of a household, where household 
sizes could vary widely; users of a park or a 
museum during a certain number of hours on 


given days, where the number might’ vary 
unpredictably according to weather condi- 
tions) ; 


- files for superiors and others interested in 
other Ministries, or international agencies; 


- inclusion as an appendix in final reports. 


At minimum 10% more than the expected number 
of completed interviews may be required, depen- 
ding on the nature of the survey, respondent 
population, sample design, and so on. As a 
general rough rule-of-thumb, 25% more than the 
expected number of interviews will usually be 
adequate for surveys of up to 3,000 inter- 
views. If the expected total sample size is not 
known a larger margin of printing over-run will 
be required. Extra copies should be printed in 
each major language version. If there are 
several minority languages 25% or so extra for 
each should suffice. 


All copies should be obtained in one print- 
ing run, as it is often costly and difficult to 
arrange second printing runs: preparation 
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charges may be incurred for each printing run, 
regardless of the number of copies; sometimes, 
a fresh original must be prepared (either type- 
set or typed); and there may be scheduling 
delays in obtaining use of the presses. 


Administering the Questionnaire 


In interviewer-administered surveys, train- 
ing and instructions must be given to so that 
each interview is as complete as possible at 
the first attempt. 


As a questionnaire for a new survey is being 
developed, or as it is being modified, the 
designer will want to review the main documents 
for training or retraining of interviewers to 
clarify what is being sought and the kinds of 
answers expected. If probing questions are 
required at any point in the survey, the train- 
ing manual will specify in detail the wording 
that should be used. The manual will also des- 
cribe the coding lists and provide rules to 
follow if the interviewer is expected to code 
the answers to the questions. 


The manual will contain details of proce- 
dural and administrative steps the interviewer 
is expected to carry out during the interview. 
Contents can include: details of the contents 
of the interviewing "kit" or bundle of supplies 
that each interviewer should receive, a des- 
cription of the interviewer assignment, rules 
on how to locate respondents, other details of 
sample-selection procedures (this can include 
some elaboration on how to follow maps, or how 
to find the exact location at which interviews 
are to start, plus the routine to be followed 
to “fund athesssecond,.. third, fourth... ...Lneer— 
view); suggestions for general deportment and 
relations with the public; procedures to be 
followed in the event of a refusal; instruc- 
tions on what to do with completed question- 
naires and other supplies at the end of inter- 
viewing. 


The interviewer's manual can be useful to 
show the question in the actual format in which 
it appears in the questionnaire, complete with 
instructions and may include an entire comple- 


ted questionnaire, as if filled out by aan 
imaginary interviewer during an interview. 
Sometimes, a dummy completed questionnaire can 


be presented with deliberate errors, so that 
interviewers can be asked to comment on how it 
should have been completed, although it is not 
a good practise to emphasize "the wrong way" as 
interviewers may be confused as to which is 
right and which is wrong. 


Visual and Audio-visual Aids 


While the questionnaire is being prepared, 
each question and each section of the question- 
naire should be considered from the point of 
view of aids or additional documents that would 
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SURVEY OF SKILLS & TRAINING 
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REQUIRED FOR HIS/HER MAIN vos? 


50D 


WOW 
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AT THE PRESENT TIME, IS...TAKING A COURSE OR PLANNING TO TAKE A 
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HIS/HER JOB SKILLS OR OPPORTUNITY FOR ADVANCEMENT IN THIS TYPE 
O OF BUSINESS? 
5OF 
Ir YES, ENTER 1 IN COLUMN “F” AND END SUPPLEMENTARY QUESTIONS 
FOR THIS RESPONDENT. 


Ie NO, ENTER 2 IN COLUMN “F” AND GO To QuEesTION 50G 


Ts THERE ANY REASON WHY,..1S NOT TAKING OR PLANNING TO TAKE A 
COURSE(S) AT A SCHOOL, COLLEGE OR UNIVERSITY? 


ENTER THE NUMBER OF THE APPROPRIATE CODE IN COLUMN "G” 


Copes FoR ITEM 50G 
50OG No COURSE AVAILABLE 
HOURS OF WORK PROHIBIT ATTENDANCE 
PERSONAL OR FAMILY RESPONSIBILITIES 
CoURSE COSTS TOO MUCH 


NoT INTERESTED IN TAKING A COURSE 
No REASON 
OTHER 


be useful for training interviewers, or to use 
during the interview process itself. The most 
common aids are "show-cards" or "prompt-cards", 
i.e., lists of answers from which the respon- 
dent is to make a choice for a given closed- 
ended question. These are particularly helpful 
if there are more than four answer categories, 
or if the answer category descriptions are more 
than one or two words each. Each interviewer 
will need a set of such cards (often identified 
by letters or colours rather than numbers, to 
avoid confusion with the questions themselves) 
with instructions for their use. Sometimes it 
may be more convenient to identify the cards 
with the number of the question they are to be 
used for, although this may be confusing if a 
given card is to be used with more than one 
question. 


Other visual aids such as drawings can some- 
time facilitate recall, for example, drawings 
showing the enumerator seated with the members 
of the household asking about household com- 
position, drawings of a typical farm showing 
the location of the farm house, land under pro- 
duction, land lying fallow, a variety of 
animals, etc., have been used successfully ina 
number of censuses. The application of the 
technique (as illustrated below in an example 
from the 1981 Census of Canada) is limited only 
by the ability of the questionnaire designer to 
translate a concept into an appropriate visual 
aid. 


Films and tape recordings of interviews may 
also serve questionnaire development or evalua- 
tion. It may be useful to try out parts (or 
all) of a questionnaire, with a recording for 
convenience of survey designers to hear how 
respondents accepted certain questions. 
Respondents' hesitation, misunderstandings and 
intonation, together with "um, er" can indicate 
questions that may be giving some difficulty to 
answer. As well, quotes from respondents' 
answers, from tape recordings, can be used in 
reports to add colour or flavour to an other- 
wise dry content analysis. However, tape recor- 
ders are relatively expensive and can easily be 
mislaid, damaged or stolen. This can particu- 
larly be a problem for interviewers, who may 
have trouble finding a place to leave a tape 
recorder unattended during field work. Adequate 
supplies of batteries, preferably of the reusa- 
ble kind, together with equipment for recharg- 
ing them, must also be obtained. 
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Chapter 4 


TESTING AND EVALUATION OF THE QUESTIONNAIRE 


Introduction 


The questionnaire can have a major influence 
on non-sampling errors and consequently on the 
overall quality and value of a survey. The 
extent and seriousness of some non-sampling 
errors will be greatly determined by such ele- 
ments as questionnaire length, language, 
sequence, flow and layout. Experience with 
surveys alone does not always provide suffi- 
cient guidance as to what question formats are 
likely to be most successful in controlling 
non-sampling errors. Where there is doubt about 
the design of a particular question or a set of 
questions, consideration should be given to a 
test of alternatives. Testing should become 
part of a formal program of questionnaire eva- 
luation. The usefulness of the program will lie 
in the accumulation of information on_ the 
nature of errors associated with questionnaires 
and in the analysis of their behaviour over 
time. The extent of testing will depend not 
only on the importance of the survey, but on 
the amount of time and resources available. 


Testing of questionnaires is undertaken to 
ensure the relevance of data and to reduce the 
risk of errors: evaluation of a questionnaire, 
on the other hand, identifies and measures the 
error which remains after testing. Thus, ques- 
tionnaire testing including its analysis is 
carried out before the introduction of a survey 
into the field, while evaluation is carried out 
during and after the collection of data. 


Testing Questionnaires 


Testing of a questionnaire is usually an 
iterative process during which successive ver- 
Sions of the questionnaire can be analysed to 
detect difficulties due to lack of clarity, 
relevance or sensitivity of the information 
sought. Questions can be eliminated or modified 
by asking them differently or positioning them 
elsewhere in the questionnaire. If difficulties 
can't be resolved through improvements’ in 
design, offending questions can be _ dropped 
altogether. 


Review of the Draft Questionnaire 


A first step is a thorough examination by 
all who will be involved once the interviewing 
has been completed. This includes’ editors, 
coders, key-punchers, programmers, and survey 


data analysts. It is a good idea to organize a 
formal discussion with some or all, once they 
have made trial attempts to edit, code, and 
capture data to see if there are any aspects of 
the questionnaire that merit revision. 

A second step in the examination of the 
draft questionnaire is a review by some of the 
more .senior and. experienced persons in. the 
Ministry/Department and members of the Steering 
Committee. The reviewers could be provided with 
a checklist of points to assist them in orga- 
nizing their comments. 


The following checklist will guide the 
review of the questionnaire. 
- concepts and definition of terms, 


- availability of information from other 
sources (relevant to the survey), 


- adequacy of the questions, 

- clarity of the questions, 

- language, 

- sensitivity of information, 

- flow or sequencing of questions, 

- length of interview, 

- coding and data processing requirements, 


- other comments. 


Review by Users 


After the internal review a third stage of 
testing involves asking a small group of data 
users to comment on the following: 


- the appropriateness of terminology in a par- 
ticular subject matter (agriculture, health, 
employment, as examples); 


- whether the reference period is suitable for 
the subject matter under consideration; 


- whether the units and ranges implied by the 
question are suitable; 


- the respondent burden; 


- the relationship between the questions asked 
and the tabulations to be produced. 


Respondent Sample 


The next step consists of submitting the 
questionnaire to a sample of respondents, 
asking them to complete it. 


Tests involving respondents are often small 
scale. They should be carried out with "real" 
respondents - that is, with individuals or 
households selected from the population of 
interest. It is essential to include respon- 
dents of different ethnic or linguistic groups, 
or regions, or sizes of community depending on 
judgements as to whether these variables are 
likely to be related to differences in impor- 
tant results in the full survey. 


Following the interview a debriefing session 
for interviewers and other field staff should 
be arranged to pinpoint the inefficiencies, 
ambiguities and difficulties associated with 
administering the questionnaire. A debriefing 
session involves interviewers and_ relevant 
field staff in discussion about field opera- 
tions from the time the interviewer knocks on 
the door to the time he leaves the respondent. 
This type of discussion generates a wealth of 
information that can be used to improve or 
modify the questionnaire. 


Typical questions include: 
- How did the interview go? 


- Did you have any problems with the question- 
naire? 


- Were there any parts that respondents didn't 
seem to understand, or that you had to ex- 
plain to them? 


- Were there any parts that required respon- 
dents to think hard, or to hunt for papers or 
documents to be able to give answers? 


- Were there any questions where you felt res- 
pondents were guessing the answers? 


- Were they embarrassed or irritated? 


- Were there any questions whose wording you 
found awkward or that used words that seemed 
strange to you, not the kinds of words or 
phrases that people commonly use? 


- Did you have any trouble 
sequence of questions, 
or instructions? 


in following the 
or the skip patterns 


- Were there any places where there wasn't room 
enough to write in the answers? 
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- Were there any semi-open or closed-ended 
questions where the list of answers was not 
complete or categories were not clear, or 
overlapped with one another, or where it was 
hard to tell which category was the right one 
for answers given by respondents? 


- Were there situations where the numerical 
answers given didn't fit the categories, or 
were given in vague quantities that respon- 
dents couldn't or wouldn't make more speci- 
fue? 


Pilot Test 


A pilot test is a final stage in testing 
procedures and it is designed to replicate the 
conditions of the actual “survey. The test 
involves all aspects of questionnaire design. 
The purpose is to discover major problems that 
exist when all aspects of survey design are in 
operation at the same time. Careful attention 
is paid to the cost of various operations in 
relation to the total cost. Equally important 
is the test of management and control of opera- 
tions. At this stage, evaluation tends towards 
overall performance rather than that of indivi- 
dual components and may cover only quality 
aspects of the questionnaire and data process- 
ing. 


Evaluation Methods 


Evaluation methods are primarily designed: 


- to provide systematic procedures for identi- 
fying errors, their sources and causes; 


- to analyse them in order to discover means of 
eliminating, reducing, or controlling their 
impact on data quality; 


- to provide a measure of their impact on data 
quality in order to assist in the interpreta- 
tion and use of the data. 


A brief discussion of evaluation methods is 
Given below. 


Observation 


In an observation program actual interviews 
are observed to detect problems related to both 
interviewers and respondents. Conditions where 
interviews are observed may not be fully repre- 
sentative of the situation where interviewers 
are left on their own. Consequently observation 
cannot consist of merely being present and 
"looking on". But observers must keep in the 
background of the interview situation not to 
bias the responses. They will need to be 
trained in observation programs and operate 
with a well planned check list of things to 
watch for. This applies whether the observa- 


tions are used for qualitative or quantitative 
purposes. 


An observation program may be expensive 
since highly qualified interviewers must travel 
to different areas to cover a subsample of 


respondents from each assignment. The sample 
size in an observation program is usually 
small, consequently quantitative data on 
evaluation must be accumulated over time. 


Qualitative evaluation data however, can be 


gathered and reported on at any time. 


Observation programs are most useful for 
continuous or repeated surveys as deficiencies 
can be ironed out over time. They can also be 
used to redirect operations in one-time surveys 
providing the survey period is lengthy and the 
observation undertaken near the beginning of 
the survey period. 


The character of an observation program is 
Similar to debriefing. If the observers are 
experienced and well trained they can evaluate 
the interviewer training as well as make judge- 
ments on the quality of the questionnaire and 
the field procedures. 


observers 
instructions on the 


To ensure an efficient program, 
should be provided with 
procedures for observation. 

The instructions should stress: 

- how interviewers introduce observers to res- 
pondents and how observers introduce them- 
selves; 


- observer's behaviour during the interview; 


- interviewer's introduction of the survey(s) 
to respondents; 


- interviewer's interviewing technique and res- 
pondent's reaction to it; 


- respondent's ability to answer the questions 
as asked; 


- length of interview; 
- deficiencies of the questionnaire; 


- any other important aspects of the question- 
naire and/or interview. 


All the information obtained from observa- 
tion should be documented thoroughly for subse- 
quent analysis. 


Interpenetrating Samples 
When the process and content of the 


interview are being tested simultaneously, it 
is difficult to separate the effects of the 
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from 
area differ- 


questionnaire on the quality of data, 
those of operational procedures, 
ences or interviewers. 


To separate these effects, 
designs such as interpenetrating samples or 
split panels can be used. If, within a given 
area: (i) a sample is selected in the form of 
two or more sub-samples according to a speci- 
fied plan; and (ii) the same sample plan is 
such that each sub-sample provides an indepen- 
dent estimate of the characteristic under con- 
Sideration; then the sub-samples are referred 
to as interpenetrating samples. The technique 
of interpenetrating samples helps in providing 
an assessment of the quality of information. 
For example, it may be of interest to find out 
whether different forms of the same question 
produce important differences in the answers 
received. It would be impractical to ask both 
forms of the questions of the same respondent. 
However, the differences can be assessed by 
interpenetrating samples of respondents using 
the same interviewers for both forms. 


experimental 


More elaborate tests of differences result- 
ing from different questions and questionnaires 
can be also carried out by various methods 
developed in the design of experiments. The 
methodology of such designs is described in 
text books dealing with sampling methods. The 
application of these methods, although theore- 
tically sound, may present practical problems. 
First of all, they are costly and time consum- 
ing. Their cost is associated with the size of 
the samples that may be required in order to 
detect differences reliably. They are time con- 
suming because the analysis of the results from 
large samples takes a long time. Thus, such 
designs should be used only in very exceptional 
cases. 


In general, purely statistical tests and 
experiments based on sampling are seldom con- 
clusive and they create practical and theoreti- 
cal problems both in the field and in the 
interpretation of results. Sampling error tends 
to dominate the non-sampling errors on account 
of the small sample sizes; for a given level of 
reliability, different sample sizes may be 
required for different elements to be evaluated 
and it is frequently impossible to select a 
random sample of respondents, due to the res- 
tricted sample size, the cost and time con- 
straints. 


Reinterviewing 


A major technique for questionnaire evalua- 
tion is reinterviewing a sample of households 
to attribute the observed differences between 
the two sets of responses to one of several 
sources: the questionnaire, the respondent, the 
interviewer, or the time lapse between the 
first and second interview. 


Cost and respondent burden restrict this 
method of evaluation to a relatively small 
sample of respondents. The longer the time lag 
between the reference period and the reinter- 
view the more likely outside events will affect 
results. A short time lag on the other hand may 
simply reproduce the original results as the 
respondents may recall too well their original 
responses. These conditions must be taken into 
account in interpreting reinterview results. 


A reinterview program can serve several 
purposes: for a one-time survey, early reinter- 
viewing reveals difficulties in questions in 
time to make changes to improve the balance of 
the survey. In a continuous survey program, it 
is a powerful tool both for the control and 
evaluation of the questionnaire. For example, 
in a continuous survey program, reinterviewing 
may establish an acceptable level of diffe- 
rences between the original interview and the 
reinterview by type of questions. Any signifi- 
cant departure from that level will serve as a 
warning to investigate the questionnaire or the 
methods of interviewing. Reinterview  cor- 
responds to normal quality control measures of 
any well maintained commercial production 
process. 


A reinterview progam can also be used to 
help estimate response bias and simple response 
variances resulting from a particular question 
or group of questions in the survey. This may 
be accomplished by the following design: each 
month, a number of reinterviews are carried out 
as a repetition of the original survey under 
similar conditions. Additional reinterviews are 
conducted by a senior interviewer responsible 
for a group of interviewers, without knowledge 
of the original results. The senior reinter- 
viewer compares his results with those on the 
original schedules, and with the reinterviews 


of the subordinate, and attempt is made to 
reconcile discrepancies between these inter- 
views. 


The accumulation of data from both parts of 
the sample will make possible a study of gene- 
ral aspects of response errors. Thus, a simple 
response variance may be estimated from the 
first part of the sample, under the assumption 
of a "repetition of the original survey condi- 
tions", and response bias may be estimated from 
the second part of the sample if it is assumed 
that the reinterviewer provides the "true" 
answers under the survey conditions. The pro- 
gram makes it possible to conduct a general 
study of the variation of interviewers' perfor- 
mance, according to interviewers' experience, 
duration of training, type of area, season of 
the year and so on. Furthermore, the field con- 
trol aspect is not neglected. The reinter- 
viewers will have the responsibility of deter- 
mining specific instances where the information 
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collected by the interviewer is 
wrong with respect to the coverage 
and when 


originally 
incomplete, 
of the household or persons within, 
reconciliation is to take place. 


Periodic analysis of the data produced by 
the program will permit detection of sources of 
errors and should lead to a continuous improve- 
ment of the general survey methodology. 


Like interpenetrating samples, the reinter- 
view program may help evaluate many aspects of 
the questionnaire. Sensitivity, . recall - pro- 
blems, effect of length and type of question 
can be reviewed by comparing for a number of 
questions the response biases and simple. res- 
ponse variances. As in the observation program, 
the costs may be large because of the spread 
and size of the sub-sample needed for reinter- 
view. Observed differences cannot always be 
evaluated precisely because of time lag (for 
example, respondents may forget some of the 
characteristics by the time a reinterview ope- 
ration is undertaken). As in a well-designed 
interpenetrating sample, a well-designed rein- 
terview program, especially for continuous sur- 
veys, may monitor many aspects of questionnaire 
design over time. 


For more details on reinterview see: (1) 
United Nations (1982b) "National Household 
Survey Capability Program. Non-Sampling Errors 
in Household Surveys". (2) Platek and Timmons 
(19 1b a 


Record Check Analysis 


It has been emphasized that evaluation of a 
questionnaire will achieve a great deal of its 
aims through detecting deficiencies in opera- 
tions which might contribute to errors in sur- 
vey data. An indication of these deficiencies 
will manifest itself in high refusal rates, 
incomplete responses, inconsistency in res- 
ponses, lack of responses, claimed unavailabi- 
lity of the information required, and the cost 
or duration of interviews. Complex experimental 
designs may not be needed to detect such defi- 
Ciencies. Rather a comparison with appropriate 
data from other surveys or a regular repeated 
review of these data in a continuing survey 
would indicate whether results are worse than 
usual. 


Many non-sampling errors such as low res- 
ponse rates, blank and partially completed 
questions may be detected from the survey 
itself. Careful analysis of the behaviour of 
such errors over time is invaluable for moni- 
toring data quality. Criteria for tolerance 
limits of certain errors will be governed by 
the type of survey, the particular type of 
error and the cost involved in eliminating or 
reducing them to acceptable levels. 


An equally important source of analysis of 
errors is the processing stage. For example, 
format, sequence, or open questions, may 
contribute to the majority of errors at the 
processing stage. Analysis of errors’ is 
particularly of value in a continuous program. 
Examination of fluctuations in the number of 
errors from question to question over time or 
the continuity of a trend would provide valu- 
able information which can be used in making 
suitable changes in the questionnaire. 


Many aspects of a questionnaire may also be 
evaluated by a study of edits and imputations, 
whether manual or by computer. If this is done 
question by question, it can be useful in 
determining the extent to which errors are 
related to the difficulty of the questions with 
regard to complexity, length, flow, order or 
sensitivity. This method of evaluation can be 
integrated with processing operations. Automa- 
tion is quite feasible, making it a relatively 
inexpensive procedure for a large volume of 
questionnaires. It is primarily useful in 
post-survey evaluation. Most of the measures 
are quantitative although a study of original 
questionnaires (especially of comments placed 
on them) may also reveal qualitative problems. 


In order to record and analyse survey errors 
systematically, special forms with headings for 
particular errors should be designed. This will 
assist in accumulating the data in a uniform 
manner and facilitate processing and analysis. 


Associated with an analysis of errors is a 
data quality report which should be produced at 
regular intervals by type of survey. For 
example, in Canada a quarterly quality report 
is produced for the Canadian Labour Force Sur- 
vey. The report provides a review of sampling 
and non-sampling errors. The sources of these 
errors are carefully monitored and an immediate 
feedback to specific operations is provided to 
control quality on a month-to-month basis. The 
report examines sampling errors and = non- 
sampling errors inherent in total non-interview 
rates by type, non-response rates by type of 
question, coverage rates, data entry errors, 
and the extent of edit and imputation. The 
Error Profile publication of the U.S. Bureau 
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of the Census also provides a sound basis for 
error collection and analysis. 


Benefits of Testing and Evaluation 


In reviewing the various methods of evalua- 
tion it can be seen that many items related to 
questionnaires may be evaluated by _ several 
different techniques. The use of these tech- 
niques is dependent on cost, time, availability 
of trained personnel and sophisticated equip- 
ment, etc. The particular programs of testing 
and evaluation have to be developed with these 
factors in mind, as well as the need for relia- 
bility if the data are to be truly useful. 


An evaluation program is particularly impor- 
tant in a continuous program of surveys. 
Evaluation of questionnaires, of the operations 
involved in data collection, of data processing 
stages, and of the results of the previous 
cycle of data collection all become input 
towards modifications of both the questionnaire 
and operational procedures for the next cycle. 
However, in order not to interrupt a time 
series which has been developed on the basis of 
a continuous survey, the questionnaire should 
not be modified on the basis of evaluations of 
just the previous cycle. Nevertheless, some 
improvements in operations procedures can be 
implemented. Experience gained from evaluation, 
after a significant number of cycles, is used 
to revise the questionnaire. Moreover, to 
smooth out the impact of the changes in the 
questionnaire on the time series or to provide 
adequate data for linkage between versions, a 
parallel run of the old and revised versions 
may be undertaken over a number of cycles. 


Feedback from interviewers and from respon- 
dents can be organized as a part of any con- 
tinuing program of surveys. Indeed, the possi- 
bility for feedback and corrective action for 
all aspects of survey operations is increased 
immensely in a continuing program. Regular 
repeated discussions with interviewers, and 
regular field visits by survey planners to hear 
for themselves how respondents react to the 
questionnaire, can be put in place as a part of 
survey procedures. 
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Chapter 5 


CHECKLIST SUMMARY OF ELEMENTS OF QUESTIONNAIRE 


DESIGN 


The previous chapters of this study have 
attempted to put forward some principles and 
practices of good questionnaire design. 


1. General planning 
of questionnaire 
development 


2. Creation of an 
organization to 
design the ques- 
tionnaire 


Has the rationale for wanting a 
survey been developed? 


Have background papers, discussion 
documents, interdepartmental and 


departmental correspondence and 
files and documents from = any 
previous surveys been’ reviewed? 


Have meetings been held with users 
to clarify objectives, needs, 
negotiate priorities and deadlines? 


Is: »there» “agreement: . on, subject 
matter, timing and desired quality 
of data? 


Has the communication process, 
internal and external, i.e., between 
all those involved in development 
and others interested, been 
designed? 


Does the needed information already 
exist? 


Have members of design team 
(resources permitting) been 
selected: project manager, subject 
matter representative, sampling, 
field and data processing experts 
and technical and clerical support? 


Have departmental and interde- 
partmental advisory committees or 
"steering" committees been organ- 
ised? 


Has an overall plan and progress 
reporting system been developed? 


Will there be adequate documentation 
of planning, carrying out and 
reporting experiences? 


As a summary guide, a checklist follows of 
many of the important elements in design of 
good questionnaires. 


This must be done in consultation 
with users. 


This can serve to identify users and 
areas of concern. 


Memoranda as required? Regular pro- 
gress reports? Meetings? Technical 
papers? 


Administrative data that have not 
been captured or analysed, or 
unanalysed results from other sur- 
veys may be available. 


Sometimes such data may not corre- 
spond exactly to what is wanted, but 
are close enough that they can sub- 
stitute for much more expensive 
fresh collection of original data. 


Includes development of schedule of 
principal events in development of 
the questionnaire. 


To help in institutionalisation of 
questionnaire design capability. 


De 


4, 


Dis 


Constraints on 
drafting of the 
questionnaire 
and supporting 
documents 


General frame- 
work of the 
questionnaire 


Question wording 
problems 
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Have concepts, topics, and units of 
measurements been decided? ~» Have 
dummy tables been prepared and con- 
cepts reviewed, to check that the 
data required for the dummy tables 
will be obtained? 


Has the survey population, length 
and timing of interview, and nature 
of interview documents (schedule or 
questionnaire or mix) been deter- 
mined? 


Have methods of selection of respon- 
dents (including sample design), 
been designed? 


Have data collection method(s), and 
skills and characteristics of 
interviewers been reviewed? 


Have the topics and sub-topics to 
be covered in the questionnaire been 
arranged by order of priority? 


Have the sections of the question 
naire been organized (introduction, 


administrative identifiers, main 
body of questions, classificatory 
data)? 


Is there need for comparability with 
other studies? 


Is each question Televant to the 
intended uses of the data? 


Are there time, cost and any other 
constraints (political, legal, 
availability of field force, and so 
on)? 


Does the first statement inter- 
viewers make succeed in "selling" 
the questionnaire to respondents? 


Is the wording simple and clear? 


Are there complex concepts’ or 
technical terms that must be used? 


Can examples be quoted of what is 
meant by a concept? 


Does a given question require a time 
reference? Is the time reference 
clear from the context (last birth, 
last job)? 


The effects of each on the wording, 
layout, etc., of the questionnaire 
must be reviewed. 


These provide some constraints on 
questionnaire design. 


Drafting in successive sections 
makes the work easier to organize 
and manage. 


The advantages in using established 


concepts and definitions, for ex- 
ample, from other agencies, coun- 
tries, or international bodies 


should be considered. 


Keep the objectives and the analysis 
Dlans firmly in view all the time. 


The introduction is vital to get 
full respondent cooperation. 


Aim for journalese - everyday words, 
short sentences few negatives. 


Definitions may be needed to be sure 
an imprecise everyday word (and most 
of them are imprecise) has only one 
exact meaning. 


Two or three neutral examples can 
help clarify a concept. Beware of 
leading or misleading examples. 


Unguided, respondents will pick 
different time references, possibly 
conflicting and certainly non- 
additive. 


5. Question wording 


problems - 
cluded 


6. Schedule vs. 
questionnaire 


7. Layout 


Con- 


full 
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Have lists of non-ordered items been 
presented in more than one order? 


Are the questions, or is the sub- 
geetye dakely to ‘béreaffected!s by 
social-desirability or politeness 
bias? 


Are any questions leading or emo- 
tively loaded? 


Is any question 
tions in one? 


really two ques- 


Do any questions aim at future plans 
or expectations? 


Are any questions’ sensitive’ or 


threatening? 


Do any questions call for recall of 
details over a long period, or for 
effort by the respondent to compile 
an answer?. 


Does the order of the questions 
affect answers? 
Does the subject matter require 


completely worded questions? 


Can interviewers handle a schedule 
approach adequately? 


Are there 
quantities or 


limitations on paper 
printing quality? 


Are there questions requiring that 
the respondent be shown something? 


Does the question sequence call for 
much branching or skip instructions? 


Can a mixed schedule versus ques- 
tionnaire method be used? 


Is it clear how much freedom the 
interviewer has in wording’ the 
question? 


Order bias in lists of non-hierar- 
chical (cardinal) items can be seen 
inpefavouring of first» and last 
items. 


This kind of bias can be reduced by 
careful wording and sequencing 
(e.g., by suggesting that the 
socially undesirable answer is quite 
acceptable) but never eliminated. 
Most questions are likely to be af- 
fected by these biases. 


Questions should avoid suggesting 
the answer or favouring one answer 
over another. 


Beware the double-barrelled dques- 
tion! Watch out for "and"! 


Projecting from past behaviour is 
generally more accurate than relying 
on respondents' future plans, which 
may contain wishful thinking. 


Careful wording and placement’ can 
help defuse such questions although 
sometimes not entirely. 


Recall periods should be kept rela- 
tively short, and not expect much 
effort to dig out documents or con- 
tact others in order to be able to 
provide an answer. Forward tele- 
scoping bias might occur. 

(a) Don't give away answers’ to 
later questions; 


(b) grouping subjects can give 
better quality responses. 
Factual questions or ones-~ with 


simple concepts lend themselves to a 
schedule approach. 


The schedule method calls for a well 
trained field force with low turn- 
over. 


Schedules make for less bulky ques- 
tionnaires (fewer pages). 


Schedules do not readily lend them- 
selves to complex branching proce- 
dures in question flow. 


The questionnaire should provide the 
complete wording (avoiding too many 
abbreviations and including punctua- 
tion), or the wording(s) should be 


7. Layout - Con- 
cluded 


8. Interviewer in- 
structions 


9. Translation 


10. Two basic types 
of questions - 
open versus 
closed 
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Are answer spaces clearly identified 
with the question? 


Do answer spaces give enough space 
for interviewers to enter’ the 
answer? 


Are tabulated answer spaces margin 
justified for questions with lists, 
or for coding of answers? 


Have lines and shading been used 
to highlight answer spaces? 


Are questions all 
numbered, or 
numbered? 


consecutively 
otherwise’ clearly 


Are interviewer instructions readily 
identifiable, separate from, but 
together with the question(s) they 
refer to? 


Is the layout (use of type faces 
and sizes, lines, arrows, shading, 
positioning of answer spaces, etc.) 
consistent throughout the question- 
naire? 


After each question, is it clear 
what question should be asked next? 


Are there blocks of questions that 
apply only to certain respondents? 


Must the questionnaire be produced 
in more than one language? How many 
more? 


Is the translation accurate? 


Is the translation too formal, not 
colloquial enough? 


For each question, would it be 
better expressed as an open-ended 
or a close-ended question? 


clear from interviewer training or 
in the interviewer's manual. Often 
interviewers must he left some 
freedom to probe or ask their own 
supplementary questions. 


They should be close to their ques- 
tions, separated from other ques- 
tions, and if meeded connected by 
lines or dots. 


For ease of tabulation. 


These visual guides help inter- 
viewers, coders and key-punchers. 


To avoid confusion in reference to 
the questionnaire. 


Instructions in boxes or in differ- 
ent type styles can be helpful. 


To minimize interviewer error. 


Interviewer instructions, verbal or 
graphic, should be plentiful. 


Filter questions (with skip instruc- 
tions) can steer interviewers past a 
group of questions that do not apply 
to a given respondent. 


A two-language questionnairecan be 
printed using alternate lines, or 
using a back-to-back ("tumble") 
format. More than two languages 
usually require separate question- 
naires. 


Back-translation helps check on 
translation accuracy. 


Colloquial terms or dialects should 
be used rather than a é correct 
standard or formal translation. 


Open-ended are good: 


- as openers, change of subject; 

- where not enough is known’ of 
likely answer classes to be able 
to draw up a closed-end list; 


10. Two basic types 
of questions - 
open versus 
closed - Con- 
cluded 


11. Coding and data 
capture 
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Is enough information available to 
draw up a closed-ended question? 


If not pre-coded, has provision been 
made for entry of computer codes 
next to answer spaces? 


Are interviewers to code questions 
during or just after the interview? 


What codes are to be given to 
“other CSPECIFY)", "no answer" and 
"not applicable" answers? 


- where responses are technical or 
complex; 

- where respondent cooperation would 
be better than to a closed-ended 
question; 

- as a source of hypotheses, and of 
wording for closed ended ques- 
tions; 

- to get more detail or elaboration 
of an answer; 

- in avoiding leading respondents 
(i.e., no answers are suggested). 


Closed-ended are good: 


- all respondents answer in the same 
frame of reference, from the same 
choice of answers; 

- variations in respondent artic- 
ulateness are irrelevant; codes 

- for capture and analysis can be 
determined before interviewing. 
Separate coding for each answer is 
eliminated; 

- easier for interviewers and 
respondents - no need to write a 
verbatim narrative answer in full 
thus quicker and cheaper’ to 
administer; 

- sensitivity of some topics can be 
reduced (e.g., income in range 
classes); 

- more complete information can be 
obtained through lists that can be 
read out; 


but 


closed-ended lists must be drawn 
up carefully, reflecting respon- 
dent experiences in categories 
meaningful to respondents; list 
must be exhaustive and exclusive 
(non-overlapping) . 


Desk research (other studies, etc.) 
and/or developmental work may be 
needed. 


For capture directly from the ques- 
tionnaire. 


Codes and space for coding must be 
provided. 


To ensure consistency, and compara- 
bility between surveys. 
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13. Administration 


14. Supporting docu- 
mentation 


15. Testing and eva- 
luation 
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Is the paper quality sturdy enough? 


Can it be printed on both sides of 
the paper? 
Have enough 


copies been printed? 


Are special shipping control forms, 
covers or envelopes needed? 


Where and for how’ long- should 
questionnaires be stored after 
capture? What is to be done with 
them after the storage review? 


Who will have access to completed 
questionnaires and under what con- 
ditions? 


Are training and procedures manuals 
consistent with the questionnaire? 
Could visual aids such as "show- 
cards" help the interview? 


Are the objectives and hypotheses 
clear enough for testing? 


Are questionnaire designers’ to 
observe or take part in tests or 
evaluation interviewing? 


Has a debriefing meeting with test 
or evaluation interviewers been 
organized? 


Has enough time been allowed for 
analysis of test results? 


Are documentation 
quate for 


procedures) ade- 
evaluation to have an 


impact on other surveys or later 
waves of a given survey? 
Are evaluation objectives’ clear 


enough for a choice of technique(s) 
to be made? 


What is to be the scope, format and 
distribution of an evaluation report? 


Questionnaires may be handled many 
times. 


Depends on interviewing conditions 
and other aspects of handling. 


To allow for unavoidable wastage, 
file copies, spare copies. 


Careful editing is needed to elimi- 
nate inconsistencies and errors. 


Some minimum level of literacy may 
be needed. 


The scope and methods of testing and 
evaluation depend on the complete- 
ness and clarity of these factors. 


Designers cannot draft question- 
naires adequately without first-hand 
experience of data collection 
problems. 


Interviewers can comment on the ade- 
quacy of the questionnaire, supple- 
menting analysis of results. 


Lip-service acceptance of testing is 
useless unless results can be incor- 
porated into a revised final ques- 
tionnaire. 
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